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CHAPTER 1 


INTRODUCTION 


PURPOSE 


The purposes of this study were to. 

1 . Examine the state of the art of mission-oriented simu- 
lation (MOS) and its use in human factors research 

T Luss the issues involved in determining the level of 
fidelity that is appropriate for the conduct of behaviorally 
oriented human factors research in civil multicrew transport 

^TTvelop guidelines for the conduct of behaviorally 
oriented human factors research in civil mu ticrew 'transport 
operations that require full-mission simulatio ( )» 

4 Develop recommendations for future researc w tic 
might fill gaps in present knowledge regarding the validity of 
simulation^ research and suggest alternative methods which 
mmht improve the productivity of such research. 

Although many of the principles we discuss have wide 
application, this study was concerned only with the ^simu^ 
tion of civil air transport operations. In this context, 1 
includes simulation of all of the stimuli presented to the 
flight crew. It includes the aircraft cockpit, visual and motion 
cues, aircraft flight dynamics, all of the aircraft subsystems, 
the flight environment (including air tra fic con ro ’ 

weather and other air or ground vehicular traffic), the cabin 
crew and all ancillary flight services (such as dispatch, ramp 
paslenger services, and maintenance). Full-mission simu- 
lation lies at the high end of the range of fidelity associated 
with MOS. The more restrictive term “line-oriented simula- 
tion” (LOS) refers to the MOS of a civil airline operation. 

PROBLEM STATEMENT 

The problem addressed was the identification of effective 
methods for developing and using MOS and scenarios that 
represent the operationally complex environment required 
for human factors research in civil air transport operations. 

A primary goal for the human factors researcher is to pro- 
duce experimental conditions that elicit behavior . 
occur unde, similar circumstances in the real world. The 
mate consideration is performance in the real world. Th 
importance of this concept in the practice of human factors 
was stressed by David Meister (1985, p. viii) when he wrote. 

...the purpose of ergonomics/human factors is to 
describe, analyze, measure, predict, and control 
the real world of systems functioning operation- 
ally (i.e., not under experimental control). 


The obvious implications of this statement for human fac- 
tors researchers, again in Meister's words (1985. p. »■■') are- 

in consequence, the ideal environment in which 
to gather data is the operational environment. It 
may be necessary for various reasons to measure 
in some environment other than the real world 
such as a laboratory or a simulator, but m sue 
cases the conclusions derived from the data must 
be verified in the operational environment. 


PROCEDURAL APROACH 

Our first step was to review the approach and the relevant 
literature and conduct field interviews with recopnrzed 
experts Next, we analyzed the information we had obtained 
nd Jiscnssed it with authorities in the use of mrssron- 
oriented behavioral research. Ou, final step was to write this 

report. 

LITERATURE REVIEW 

The existing literature was screened for information 
related to MOS and the development of scenarios for e av- 
ioral research. Although NASA has used MOS for training 
since the beginning of the space program and l«Vr a.rlm 
began exploring the training potential of FMS in the 
1970s, little has been reported on its use for human ac ors 

Few FMS studies have involved civil aviation operations. 
One of the few was the pioneering study 
NASA Ames Research Center in the mid ( 

Smith 1979) Two more FMS studies were being one y 
researchers at NASA Ames while this study was being con- 
ducted f Foushee, appendix A, and Murphy, appendix B). 

The,'. 2 wis a paucity of a.»di« dealing with ihe pre- 

dictive validity of results achieved using MOS for research. 
Most of the reported studies were concerned with the perfor- 
mance in a simulator alone, or with the transfer of Uami g 
The transfer-of-training studies were not particularly he lp u 
because while they can predict the amount of simulator 
trainin ’ that will contribute to real-world task performance 
they do not necessarily reveal the real-world validity of 
behavior that occurred. 

Unquestionably, there are common elements among 5 am ' ' 
lation requirements for training and for research. However, 


there are important differences. These differences arise not 
only because research objectives and training objectives are 
liferent, but also because behavioral research imposes more 
stringent research requirements for the control, measure- 
"tent, and generalisation of results. With the exception of the 
publications of Lauber and Foushee (1981) dealing with the 
use of FMS in High. „cw .„i„ ing , « fj d , h „ ,™' t ' 

tant issues of scenario development for FMS have been given 
only cursory attention in the literature. 

Despite these difficulties, the literature search was helpful 
While training was the purpose of most of the reported MOS 
and LOS work, the training literature addressed many 
scenario issues that are relevant for the design, development 
and execution of simulation scenarios for research. 

For example, the training literature stressed the impor- 
tance ot a multidimensional concept of fidelity. “Fidelity” 
has been described in many ways. It defies simple description 
or measurement and. unfortunately, there are no easy 
answers to questions of fidelity requirements for research 
(see appendix C). It also is clear that there are controversial 
issues regarding the role of physical fidelity in simulation - 
considering such areas as the importance of platform motion 
low-amplitude vibration, realistic control loading, and visual 
system fidelity. Here again most studies have dealt with the 
e ects of these variables on performance in the simulator or 
on transfer of training. The predictive validity of a simulator 
performance on real-world night operations is far from clear. 


FIELD INTERVIEWS 


Field interviews with experts in the use of MOS were 
particularly helpful. Time and budget constraints prevented 
our contacting all knowledgeable people in each area of inter- 
est, and in some cases we were restricted to telephone inter- 
views. However, because we were able to selectively interview 
recognized leaders who were well informed regarding activi- 
ties in their respective fields, we were able to take advantage 
of the insight and practical knowledge of many of their 
fellow researchers. 


REPORT SCOPE AND ORGANIZATION 


This report is concerned solely with the use of MOS for 
luman factors research in civil air transport. Its intended 
audience ranges from qualified researchers who may not be 
ully familiar with the FMS of the air transport environment 

, C °T-rr atC ° r lnstItutlonal research managers who have to 
make difficult judgments regarding which research programs 
to support. 

The areas discussed range from the myriad of conceptual 
issues which have to be considered and evaluated when plan- 


ning to the pragmatic lessons learned from past FMS studies 
At the onset of the study, we were asked to concentrate on 
the appropriate use of FMS, on considerations of scenario 
design, and on their implementation for research purposes. 

e were asked to avoid such other issues as requirements for 
motion and visual systems. 

While this report assumes basic training in behavioral 
research including familiarity with experimental design and 

lneth0d 1 S - more tllan traditional training is required 
r rcsearc 1 . A critical domain of information lies 

between the lessons learned in basic behavioral research train- 
ing and the requirements for effective FMS research in 
aviation. It is information that has been learned on the job 
by those researchers who have had to perform applied 
research in complex simulation environments. 

As expected, clear answers to the critical issues were not 
always available because often there was a conflicting inter- 
dependence among considerations that outwardly seemed to 
be well-defined and separate. Resolution of pragmatic prob- 
lems which were created by the desire to study behavior in a 
simulated environment usually required reasoned trade-offs 
and judgments by highly skilled researchers. It was not 
always possible to reduce such judgments to a firm set of 
principles and procedures. Moreover, in many cases, time and 
udgetary limitations dictated the boundaries of what could 
be considered in a simulation study. Despite these difficul- 
ties, our goal was to identify the basic issues involved in plan- 
ning applied behavioral research utilizing FMS, discuss the 
wi e gamut of considerations that must be evaluated and 
provide guidelines for the use of FMS. 

Chapter 2 discusses the fundamental issues and require- 
ments of applied behavioral research for which MOS is the 
research vehicle. These issues include problem definition, 
s udy-plan development, experimental strategy, test prepara- 
tion data collection, data analysis, and interpretation of 
results. Each is important, and each requires specific consid- 
eration in all behavioral simulation studies, whether they 
involve part-task or FMS. 

Chapter 3 then narrows the scope of discussion to the 
development of guidelines for doing full-mission LOS 
research. Topics include the foundations of LOS when it 
should be used for research, research team composition, 
research subject selection and training, scenario development 
scenario script writing, operating team training, scenario test- 
ing and running, subject debriefing, and lead-time considera- 
tions. A preimplementation checklist as a summary of the 
chapter is included. y 

Chapter 4 contains research recommendations to fill gaps 
in knowledge regarding the validity of simulation research 
and considers alternative methods which might improve the 
productivity of such research. The recommendations discuss 
testing full-mission-research validity, studying alternative 
forms of simulation, optimizing mission-oriented research 
developing human-performance models, identifying subjec- 
tive measures of fidelity, and integrating research efforts. 
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The appendices are particularly important. Appendix C 

discusses the fundamental issues and requirements of applied 

behavioral research. It discusses the general 
ence how and why applied behavioral research is conducted 
the basic responsibilities of the researcher, the three genera 
purposes of applied behavioral research, the application c 
ext of the research as it changes with advances in echnol- 
o?y. tL choice of research vehicles, and the fidelity and 
validity of research simulations. All of these aLtors ar 
involved in the rational determination of the researc r v c 
Z environment required for effective, behaviorally 
oriented, human factors research and is true 
whether the requirement is for relatively sunple labora 
•moaratus, part-task simulation, or a sophisticated FMS. 

The field interviews, documented as trip reports, wer 
converted into the eight “case studies” in appendices A . 
and D through I. These case studies range fron Conrad 
Kraft's narrowly focused and classic study of visual illusions 
dmit night approaches, through studies involving marine 
S handUng. and a variety of evil and military aviation 
pph tLs. These studies offer the insights of experienced 


and successful investigators who have had to resol 

issues discussed in the remainder of this report. 

The authors are grateful for the time and insights provided 
by the investigators we interviewed during this stu y. 
worthy contributions were made by Dr. H. Clayton Foushe 
If NASA Ames Research Center; Dr. Thomas J. Hammell 
and Dr Mvriam W. Smith of the Eclectech Associates Divi- 
sion of Ship Analytics, Inc,. Dr. David D. Woods of the 
Westinghouse R&D Center; Mr. Thomas O V 1 
Roeimi Aerospace Company; Dr. Richard b. Edwards oi 
Boeing Computer Services; Dr. Conrad L. Kraft of the Boeing 
Company (Retired); Mr. David Shroyer of United Airlines 
Training Center; and Dr. William J. Cody of McDonnell- 

D °^:: r ^i"r;uidance provided in meeting 

with Dr. David C. Nagel. Dr. Charles E Billings D, .John K. 

Lauber, 1 and Dr. H. Clayton N f ^ee nd 

Research Center. And we thank ^ K Center and 
Mr Robert T. Shiner of NASA Ames Research Center and 

Dr.' Richard M. Frankel of Wayne State University for their 

thoughtful review ol the drat t manuscript. 
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full-time member National Transportation Safety Board. 



CHAPTER 2 


PLANNING AND CONDUCT OF 


SIMULATION-BASED RESEARCH 


INTRODUCTION 

This chapter discusses the basic issues involved On appUri 
human factors research, including lhe JPP f '^ applied 

It':! resTaml- rashs from stud, planning <0 dara mren 
pretation and in it the assumption is made that a PP lied I ^ 
research is to be conducted to solve an operational or systen - 

^Wilhg^lnd Mills (1982) compiled a catalog of methodo- 
logical considerations for human factors research in systems. 
As part of their work, they derived a seven-stage classifica- 
tion^for drawing together the methods and procedures apph- 
hip to conducting simulation-based researc 
specific system-design questions. The seven stages of planmng 
and execution for human factors research ar . ) 

Definition. 2) Study Plan, 3) Experimental Strategy, ) T 
Preparation. 5) Data Collection. 6) Data Analysis, 

7) Interpretation. 

PROBLEM DEFINITION 

The first step in any research program is to fully under- 
stand the practical problems to be investigated and the infor- 
mation that will be needed for the user of the research result 
to make an intelligent decision regarding them Tte pro c 
statement must be defined in terms of researchable and 
able questions that are operationally meaningful, 
stage .^a subject matter expert (SME). although not necessar 
ilv working for the researcher, is a critically > m P or a 
resource for developing - or negotiating -the 
the problem statement. The SME can also be of 
help in establishing the form of the interpreted results 
will be most meaningful to the ultimate user. 


terms that presuppose the approach necessary to answer the 
practical problem. For example, the user may ask whether 
pilot fatigue in a given situation will affect ability to perform 
normal flight tasks. The issue behind the question can 
crew scheduling and duty hours involving not only basic 
safety issues, but also questions such as the number of crews 
necessary to sustain flight operations It is dl tlcU t l ° 
the right question without fully understanding the opera 
tional problem; thus, the researcher must learn the nature of 
c operational problem that led to the practical question to 
teZ that the research question reflects real-world needs. 


Establishing the Research Objective 


Establishing the Operational Need 


Frequently the operational user of the research results will 
frame the research question in practical terms. Tin ca 
.rente a very real problem for the researcher because if the 
baJic problem is not carefully examined, the research ques- 
tion^may be framed in terms that provide incomplete or erro- 
rs miswers. The research question may be framed 
terms that do not include all of the important issues, or m 


Once all aspects of the problem are understood i the prac- 
tical question must be translated into an answerable for . 
£ first step is to determine what relevant in^—, 
available At this point the researcher s job is to secu 
information - not to do empirical research. Research may be 
required to answer the operational question, but this shou 
not be an automatic assumption. Frequently tie retire 
infmma ion can be obtained by a literature review and dis- 
cussions with others who are knowledgeable in the problen 

art If it is established that additional research is necessary to 
answer all or part of the practical question, the quesUmi 
must then be translated into testable form. The P ract *f 
question must be stated as a research question and framed m 
terms of specific relationships among measurable variables 
(Cody 1984). If the factors of interest are not direct) me 
surable they must be defined in terms of other variables tha 
are This definition process may again require the use o 
SMEs co be sure that answering the research question will 

answer the practical operational question. 

Often the practical problem is stated in terms that are 
undefinable from a research standpoint. The earlier examp e 
of a ouestion about the relationship between fatigue and 
normal flight tasks illustrates the difficulty. Despite a great 
many attempts, there is neither a widely accepted definition 
of fatigue (Bartley, 1976), nor an industry concensus on 
define ion of such basic concepts as cockpit work oad^ 

If ihe users are not likely to see the relationship between 
the planned research and the operational problem, a user 
education program will be needed. Tins need is impm anUf 
the operational problem, which has been raised by fi e user 
is no’ directly answerable. Good communication with the 
user at this stage of development is the only way to minimue 
problems of this nature. 
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In summary, the researcher’s main task in the problem 
lnition stage is to collect and organize the relevant facts in 
r er to explicitly determine what information is needed to 
s ate the practical question in answerable form A study plan 
d„r, should be prepared ,ha, clearly s.a.es "vZ £ 0 be 

eu-hIM IS ,C i AI lhe SiH1,e , ' n,e l * le resear cher should he 
stabhshmg al least a general unde, standing of the form of 
h end product and thinking „f ways t0 ‘ tilte „ ° f 
that are meaningful to the user population. 


STUDY PLAN 


I ) £m?!T " StU<l | y Pl " lndudes lh ' four tasks: 

iron , e r r oach - 2) ,he 

conditions 3) define the variables and develop the prelimi- 
nary scenario, and 4) perform a sensitivity analysis. 


Framing the General Approach 


At this stage, particularly if the research involves explora- 
orv 01 lypothesis testing, the research project should be dis- 
c ssed w„h others who have different interiors and expe tise 

a Zr 0 l" ,ide differe " CCS ° f ° Pmi ° n ™ 

approach. Other investigators may see the research problem 
and approach differently. On the other hand, in the^ase of 
evaluative research, fundamental differences of opinion 
about the approach are less likely. For a discussion of the dif- 
rences between exploratory, hypothesis-testing, and evalu- 
ative research, see appendix C. 

1 3rCher alS ° Sh ° uld be cau «>ous not to let knowl- 
ge o t e availability of particular research vehicles espe- 
ally large simulators, be an overriding influence on what s 
equmed to support the research. There always is the tempt" 

ion for the equipment available to determine what the 
research vehicle will be. 

It is worth remembering that the nature of the research 
problem may offer an opportunity to structure the research 

dia?n 5 r a n cflT tl0n hr hlCh h3S applications be y°"<» ‘he imme- 
knowl 2 u ^ m 3nd ° 0n,ributes to general scientific 
knowledge. Moreover, a research project that solves a practi- 

cal problem by developing a genera] principle almost invaria- 
bly provides a better answer than research which does not 
t Im another advantage. If either the characteristics of the 
sys em being investigated or the operational problem changes 

on am 'vl d T* ^ f <tUdy ° f S™" 1 ^ciple, should 
remain valid, whereas the results of a study framed in too 

restrictive a manner may no longer be applicable. 

A good example of research that both solved a practical 
problem and made a substantial general contribution is the 
er.es of experiments performed by Kraft (see appendix G) 
o etermine whether characteristics of the B-727 aircraft 
were comribu.ing a series „f la „ ding . shor , ac ™ 


during nighl-landing approaches under visual conditions His 
early analysis found commonalities among the external visual 

"Tr ,,h inCid “' S - «*„, on “ e 
hat the underlying reason for the low approaches was not a 

characteristic of the airplane. The basic cause was a visual 
l usion of height produced by the combination of lighted 
and tilted terrain behind the airport and the lack of any 
hghts in the foreground - a condition frequently found dur 
ing night approaches over large bodies of water. ’ 

SOlUti ° n f0r thc P ro blems associated with 
, night approaches was an educational program for pilots 
and a revision of cockpit procedures. If a Z killfo 
researcher had undertaken the research, however and 

teristks of the° n the e™ ° C ” did the cha ™c- 

tenstics of the aircraft contribute to the accidents?) the 

answer might well have been equivocal. And even if the prac- 
tice problem had been solved, the more important gen a] 

contribution would have been lost. am general 

Kraft’s discovery of thc underlying cause of this series of 
ZllT ntlr hettCr UndCrStandi "S « fundamental 

a major contribution to aviation safety and 

Wickens’ 7’] 984) “f* ^ 3Pplied behavioraJ research, 
kens (1984) work on a multiple-resources model of 

Human performance and aircraft-design display is another 

excellent example of theoretically guided research performed 

at the same time a specific problem was being addressed 

The researcher s conception of the research problem - as 

either narrow and specific, or as an example of a broad and 

general problem - determines whether thc study may have a 

potential value beyond the immediate solution of the prob- 

, S 65 not suggest that ever V applied research pro.ect 
an be expected to contribute to fundamental scienTfic 
nowledge. We do suggest, however, that whenever p7s hie 
t behooves the researcher to choose a research settled 
conditions that permit the examination of underlying causes 


Describing the Operational Conditions 

Once a decision about the general research approach has 
been made the study plan should be developed The fim 

S,ep ,s “> description ot ,L „ ™ 

cond, ,„m of i„, cres ,. Describing , he p|m ^ 

wi P |',c ' dep " ldln f °" »'« marcher's familiarity 

the operational environment, the scope of the problem 

.7^77^1, y - DeSCribing thC 0perati0 " al c °7fofo^7of 

interest with precision is an important step and it should be 
done with considerable care. “ D 

eXMrieneeTn^he SllOU, ‘ , ' ffon ,0 6“" f'«t-ha„d 
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Defining the Variables and Developing the 
Preliminary Scenario 


Armed with a general approach and description of the 
operational conditions, the thud step in the development o, 

t of m number and values of the independent varia- 
ble conditions to be held constant, and the data needed 
to derive the desired performance measures. ^ Also , at thi 
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performed, q should result in a con- 
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ional relevance. The decisions of the researcher at this stag 
start the formulation of the preliminary research scenario. 
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Performing the Sensitivity Analysis 

After developing the preliminary scenario, the researcher 
should do a sensitivity analysis (Cod* ^ 984) by thinking 
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the study, including estimates of the number of run, 

^ Ulred , the number of objects needed, and the amount of 
da to be accumulated. All of these issues have practical, as 
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in many cases, there is not a theoretical basis for 
measurement. 

The best course of action is to examine what has been 
measured in previous studies, listen carefully to SMEs and, if 
possible, discuss measurement issues with other investigators 
who have dealt with this problem for similar tasks. These 
activities should define candidate measures, which should 
then be tested empirically to determine if they produce per- 
formance differences which are detectable by other means, 
such as (but not limited to) ratings by expert observers. 


TEST PREPARATION 


At this stage, the researcher should have a well-developed 
study plan and can now start thinking seriously about the 
many practical matters that will need attention prior to the 
actual conduct of the research. The practical matters include 
determining simulator capabilities, resolving any conflicts 
between simulator capabilities and research requirements, 
coordinating with simulator support personnel, developing 
the detailed scenario, establishing procedures for the conduct 
of the experiment, recruiting subjects, training subjects, and 
pretesting all equipment and procedures. 


Simulator Capabilities 

In most cases, simulator facilities are constructed to sup- 
port flight crew training, engineering studies, or broad, long- 
range programs of research. One of the researcher s first tasks 
is to discover the specific simulation capabilities that are 
available, and to determine whether or not modifications can 
be made if they are necessary for the proposed research. 

An early visit to the simulator facility may be sufficient 
to discover if there are conflicts between the research 
requirements and the simulator’s capabilities. This initial visit 
should be a prelude to several meetings with facility person- 
nel to gain a thorough understanding of the simulator fea- 
tures, its operation, and the duties of each member of the 
simulator support staff. It is important to establish a sched- 
ule of meetings to work out the details of preparing for the 
study and a schedule for preparing the simulator. Individual 
responsibilities of the personnel involved should be deter- 
mined at this time. 

Conflicts Between Simulator Capabilities and 
Research Requirements 

Conflicts between simulator capabilities and research 
requirements usually involve details rather than gross discrep- 
ancies when modern simulators are being used. For example, 
the researcher may want a particular event to occur contin- 


gent upon the occurrence of one or more other events. While 
the simulator may be capable of producing the event desire , 
it may not be able to produce the event at the desired time 
ot under the desired conditions. This sort ot conflict may not 
be discovered until the final scenario is established, and the 
programming and other work is under way. 

It is impossible to list all of the possible problems asso- 
ciated wi-h the capabilities of the simulator that may be 
encountered. Many will be discovered during preliminary 
testing; others may not be discovered until the actual data 
collection. To discover as many problems as possible, the 
researchei should become familiar with the operating details 
of the simulator early, and should plan upon spending con- 
siderable time with the simulator support personnel review- 
ing details of the research requirements. 

A very common problem is whether to modify the 
research program or the simulator. At the conceptual level, 
the research program is normally within the control of the 
researcher. However, the simulator can be under the control 
of a higher level of management, or another division of the 
organization. While it is usually easier to modify the research 
plan, in some cases modification of the simulator may be 
necessary. Unless the simulator characteristics are known 
well in advance, and the need for the modification antici- 
pated, the cost and time required to modify the simulator 
can be substantial. An additional complication may be that 
the simulator is also being used for other research or training 
the needs of which are not compatible with the proposed 
modification. 


Coordination with Simulator Support Personnel 

Experience has shown that the researcher cannot always 
assume that a single person will be the key simulator facility 
contact to coordinate all aspects of the preparation, schedul- 
ing, and operation of the simulator. Usually several people 
will be involved and the researcher must be prepared to coor- 
dinate their activities. The simulator support personnel will 
not know what is important to the research project or w rat 
must be done, unless it is spelled out for them. (See Way and 

Edwards in appendix F.) 

The researcher may have to work with several specialized 
support people, and should make every effort to understand 
their responsibilities - the things they can and cannot be 
expected to do. If physical equipment needs modification, 
equipment engineers and technicians will be involved. 

At this stage, it may be necessary to consult as many as 
three programmers regarding varying aspects of computer 
control. Normally, a real-time programmer will be responsi- 
ble for the software controlling the simulation, the sequence 
of events, and the control that can be exercised at the con- 
sole. If a computer-generated-image visual system is used, a 
visual-scene data-base programmer will be needed to perform 
any changes or additions to the visual scene. Changes in 
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visual scenes are described easily in terms of objects to be 
portrayed, visibility characteristics, and visual events, but 
even simple sounding modifications can take days or weeks 
to program. 

A third type of programmer may be needed for data 
acquisition, if there will be event- or time-contingent sam- 
pling, or processing of data between capturing the data and 
recording it. A special programmer for data analyses may also 
be needed subsequent to the test runs. The researcher and 
the last two types of data programmers should jointly plan 
the data collection and analysis methods. Ease of data analy- 
sis can be enhanced greatly by the way data are labeled and 
organized when collected. 

The researcher should know enough about the console to 
operate the simulator if necessary. This means that he or 
she will have to become acquainted with the operation of the 
control console, and will be able to set initial conditions and 
invoke certain options. This is particularly important for 
checking that the right conditions have been set for different 
trials. It is very easy to set the wrong sequence of trials if 
multiple sequences with different conditions are involved in 
the simulator runs. 

Various technicians will be responsible for operation of 
major subsystems, such as the motion base and the visual 
system. The researcher should have at least a general famil- 
iarity with the duties these people perform, because should a 
fault occur, he or she would know who to turn to for help. 

Scenario Development 

The scenario creates a simulated real world for the flight 
crew, and creates the events of the flight to be observed and 
measured for the research purposes. The scenario establishes 
the mission objectives, tasks, environmental conditions, event 
timing, and rules of operation. 

As we have discussed earlier, scenario preparation should 
begin while the study plan is being developed. Because simu- 
lator capabilities are critical, some modification of the 
scenario may be necessary. At this stage the researcher 
should be concentrating on refining the scenario to be sure 
that it is consistent with actual operations. Specific identi- 
ties, values, and occurrences need to be assigned to the pre- 
vious, more generally defined characteristics of the scenario. 
One or more SMEs (e.g., an experienced crew member for 
the type of aircraft simulated, or an experienced controller 
tor the ATC areas) will be needed to establish these details. 

It the same subjects are expected to participate in more 
than one test run, the researcher may need to develop varia- 
tions on the scenario to create apparently differing test con- 
ditions. The problem is to maintain scenario equivalence at 
some level of conception for all conditions considered to be 
constant, even though those conditions may appear to the 
subjects as different. Cody (1984) was able to overcome 
this problem in a study of low-level navigation and attack 
missions of military aircraft. 


Cody s flight path involved several waypoints and multiple 
targets. He constructed a template of the full flight route and 
rotated it several degrees for each variation of the scenario. 
In effect, all of the relative values of flight-segment length 
and turns remained the same, but the absolute values of all 
headings changed. The flight legs were long enough that the 
pilots did not see the constant pattern in the various mis- 
sions, therefore each scenario appeared to be unique. 

Woods (1984) suggests that if the class of behavior of 
interest is well defined, many specific instances of members 
of that class can then be regarded as equivalent. For example, 
it is possible to create a variety of decision problems as long 
as they meet the criteria defined for a class of decision prob- 
lems. If this rationale is to be applied successfully, the 
researcher must have a valid theoretical basis for defining the 
behavioral class, and the simulated examples of the behav- 
ioral class must have operational relevance. 

This line of reasoning again argues for the importance of 
translating practical problems into a more theoretical context 
to seek generally useful solutions. However, if the nature of 
the practical problem excludes any basis for establishing 
equivalence between expected behaviors, the researcher has 
no choice but to use new subjects for each variation in the 
scenario. 


Procedures 

The execution of any MOS is complex. Extensive research 
and support team coordination, teamwork, and practice are 
required to execute the simulation properly. In this context, 
procedures are the organizational, management, and oper- 
ational plans and schedules for the data collection. 

The procedures should outline the responsibilities for each 
of the members of the research team and support crew, and 
should include contingency plans for potential problems, 
such as a simulator failure. Logistic needs, such as preflight 
briefing materials and training rooms and equipment, should 
also be addressed in the procedural plans. A checklist for the 
initial state of all simulator switches and control settings is a 
key procedural document. 

The subject testing schedule, the training subjects are to 
receive, and the amount of practice to be permitted are an 
important part of the procedures (Williges and Mills, 1982). 
The training programs and the performance criteria to be met 
at the end of training must be developed, and the supporting 
documentation (e.g., questionnaires and forms for keeping a 
log of the test runs and for recording observational and sub- 
ject biographical data) should be prepared as part of the pro- 
cedures. If collateral testing of the subjects is part of the 
research program, the protocol to accomplish it should also 
be developed at this stage. 

An overall schedule for the pretesting and data collection 
phases will need to be prepared. Time allowances in the 
schedule should also be made to include the daily checking 
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is rarely the case even with the best of planning. Simulator 
faults, unanticipated subject behavior, the wrong setting o a 
condition, or a host of other “gremlins” ca « u P set even 
best plans. Additional time and subjects should be allowed to 

take care of test-run problems. 

The most serious problem that can arise is either an undis- 
covered error in the setting of a condition, or a fault in one 
of the automated data-collection systems. A mandatory pre- 
caution is to run the data through a “quick-look program 
on the same or the next day after it is collected to be certain 
of its integrity and reliability. Otherwise, it may be days or 
weeks before the problem is discovered. Depending upon the 
variables manipulated, this check of the data can also con- 
firm that the scheduled conditions actually occurred. 

It is most helpful if the simulator has the ability to get 
hard-copy “snapshots” of selected CRT displays. A snapshot 
of a formatted display, taken at the beginning of each mis- 
sion trial, showing the status of conditions that are prepro- 
grammed. or set at the simulator console, is a reliable record 
of the condition settings. 

Individual subject differences, i.c„ those unaccounted tor 
by study factors, almost always contribute more to the vari- 
ance of the data than any other source. Data from a test that 
measures an ability or characteristic thought to be related to 
the behavior of interest in the simulation test can be use as 
a covariate that may account for some of the individual 
differences. For example, a vision test might be a valuable 
source of collateral data if the behavior of interest is visual 
search or the reading of a display. As another example, sub- 
ject age was found to be the major source of variance in an 
automated training study (Vreuls et al., 1975). If age ata 
had not been taken and used as a covariate, the independen 
variables would not have produced a statistically significant 
performance change. Questionnaire data on experience and 
other biographical factors could also provide collateral infor- 
mation that can often be helpful in sorting out the test data. 


DATA ANALYSIS 


The data reduction and analysis procedures should be 
planned ahead of time to minimize the time and effort 
required during data collection, and to assure their relevance 
to the research and practical questions. While most research- 
ers will be familiar with basic data-analysis concepts, if he or 
she docs not have a high level of data-analysis competence 
an expert should be used during the planning and analysis o 
this critical function. The researcher also should be prepared 
to do additional, unplanned analyses. Frequently, an 
unexpected facet of the data will suggest additional analyses 
and may lead to a better understanding of the research 

^Tf'the research is directed toward system development or 
improvement, the data are usually analyzed in terms of indi- 


vidual, crew, and system performance. Inferential statistical 
tests can be used to establish the reliability of effects found, 
and confidence limits on the data. If it is appropriate to the 
study design, the percentage of variance accounted for is a 
useful indicator of the relative effects of the manipulated or 
measure d variables and their interactions, and of other ana- 

lytic concepts. s . 

A recent review by Rouse and Rouse (1984) o a mos 
200 evaluative studies of complex human-machine systems 
was pci formed to analyze the degree to which definitive eval- 
uative results were produced as a function of factors sue 
the research vehicle used, the type of study, the type of mea- 
surement and the domain of the study (vehicle control, pro- 
cess control, maintenance, and so forth). The authors con - 
concluded with this general principle: “One is more likely to 
obtain definitive experimental results if the method chosen 
allows a reasonable degree of control, and the measures 
chosen allow fine-grained analysis of performance.” 


INTERPRETATION OF DATA 


Once the analysis of the research data is complete a 
remaining and vital step is to communicate the results to the 
customer or user community in a form that is likely to 
resolve the problem that was the reason for the research. If 
the data arc to be used for systems design, the results should 
be in a form that permits engineering trade-off comparisons. 
Other applications may call for general, or composite presen- 
tations (Willigcs and Mills, 1982). 

It the study was able to produce findings that can be gen- 
eralized to a class of problems, or is a contribution to a body 
of scientific knowledge, the researcher should publish the 
results. Regardless of the outcome, if the study involved a 
large scale simulation, the researcher should describe the 
planning and execution of the study, the resolution of any 
problems encountered, and the lessons learned. The lessons 
learned should include both the things that went particularly 
well and the things that didn’t - i.e„ those things that the 


CONCLUSION 


This chapter has discussed the process of applied behav- 
ioral research in MOS. The research process has been well 
organized by Williges and Mills (1982). and the issues dis- 
cussed are familiar. Researchers or research managers who are 
new to FMS research should consult the source documents 
for a more thorough presentation of these issues than could 
be presented in this overview. A more fundamental founda- 
tion for why and how applied behavioral research is con- 
ducted is presented in appendix C for the interested reader. 


13 




CHAPTER 3 


guidelines for line-oriented simulation research 


INTRODUCTION 


Performing LOS research, which satisfies both the w 
tific and operational communities, is a complex and difficult 
task Fortunately, many of the difficulties can be alleviated 
by following the basic principles discussed in chapter 2 an 
m appendix C.^ The purpose of this chapter is to combine 
those principles with the lessons learned from LOS ^ inl " g 
and research experience, and present them as guidelines 
the use of LOS in applied behavioral research. ^ 

Line-oriented simulation is a specialized typ • 

mission is a civil air-transport operation. Line-onente simu- 
lation uses sophisticated flight simulators and a detailed and 
faithful simulation of specific operational segmeni * to P 
vide an environment that has many of the attributes of 
real world of civil air transport. When LOS is used or tui 
factors resea, cl,, the objective of this hrgh le.el of Mr 
fidelity is to produce performances equivalent to the per 
formances similar individuals would produce after receiving 
similar stimuli in real-world operating conditions 

A basic assumption of LOS is that the more faithful the 
simulation of real-world stimuli, the interface, between i d - 
vidual crew members, the systems they control, un^he sy 
terns that influence and regulate then behavior (e.g.. ATC, 
Federal Air Regulations, and company regulations an 
cedures) the more likely it is that the behavior achieved in 
the experiment will be the behavior that would be produced 
under similar circumstances in actual line operation^ 

A high level of fidelity has a further and practical ad 
tag e It increases the “face validity" of the experiment in 
eyes of both the participants in the experiment and in the 
potential users of the research. This is important because the 
more face validity the simulation has. the more wjfifcn* 
the ultimate users in the aviation community are likely to 

have in the experimental results. however 

A basic difficulty for researchers and users alike, ■ 

is that regardless of the level of fidelity achieved, one canno 
be certain that the behavior observed during the experimen 
is anything more than the behavior in the simulator of 
particular individual o, nigh, crew at that particular .me. 
Individual performance varies even unde, similar conditions 
in the real-world, and there can be even greater differences 

2 * i • r rnntnins un overview of the more fundamental pnn- 
ing the fidelity and validity of research simulators. 


among individuals. As we noted in chapter 2. "individual sub- 
ject differences. ..almost always contribute more to ie 
ance of the data than any other source. ’ 

A further complication is that, despite signi c 
advances in simulation technology and LOS scenario devd p- 
ment. real-world operations cannot be rep^uo* x - 
The researcher, therefore, can never be certain that the see 
"duces all of ,he cues that ^ 

dictive validity of the experimental 
little ha d evidence to cither support or refute t ^ 

contemporary state-of-the-art LOS produces the same flight 
crew behavior that would result if the simulated sftuaUo 
occurred in real flight- Significant improvement* ^m aUph « 
of LOS however, have resulted in an important consensus 
* the aviation community. The ™ ^ 
only does modern LOS simulate real-world n.gh c fec J 
but more importantly, it produces the equivalent of a 
world behavior. The balance of this discussion assumes 
the industry consensus is correct. 


FOUNDATIONS OF LOS RESEARCH 

Despite more than 30 years of Government-sponsored 
research using flight simulators, and 

military and commercial airlines using simulators for fliglU 
training and checking, line-oriented FMS is a relatively new 
, lt began with Northwest Airlines pioneering 
attempts what is uow known as Ime-oticnted night » 
i | P OFTt and with Ruffed-Smith's simultaneous use of 
FMS fo, Latch in bis landmark study (Ruffcll-Stmth. 
,97,1 a, NASA Ante, Research Center Since then^uch 
material has been published regarding the use ot LOS for 
traili ng. Unfortunately . very little documentation is available 

regarding its use for research. 

Commonality of LOS for Training and for Research 

L ine-oriented simulation used in research and LOS used in 
training have much in common Both .tac t he same genera 
rules for scenario construction and execut un. Both stress a 
high degree of realism and meticulous attention to details 
simulate all of the important elements and interactions of an 
aHil operation. Much of this material is derived from 

LOFT experience. 
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preceding page blank NOT filmed 




Without question, the foundation for this chapter is the 
Lauber and Foushee (1981) Guidelines for Line-Oriented 

LOFT rTo^ ' ," iS , ' rlS ” , “ d ™ s 
todav Z i n? PnnClpleS they delineated remain valid 
>• ie LOS principles outlined in this chapter differ 

somewhat because of their orientation to research rather 

search ° LOS '"f 7° reflect the results of our literature 

arch. LOS studies which have been conducted since the 

Lauber-Foushec report was written, the case studies reported 

FMS expens and'' P6rS ° nal comn,unication * with recognized 
^ ex P e ns, and our own experience. 


US c ~' ab ' e ' - lf ** performance become, 

,„Z h sZ:7;r bcine ,i,ey hmc * w 

evcrylhing ,t V T “l”"’' and can be «P««ed to do 
y mg they can to achieve a successful result There k 

nothing approaching this personal identification for a volun 
teer pilot in a research simulation 


WHEN TO USE LOS 


Differences Between LOS for Training and 
tor Research 

FMs 7 'Ti ted simulation is concerned with the use of 
research' training , Here we are concerned with FMS for 

research the 1 16 ob ) ectlve is to change behavior. In 

. ■ objective is to obseri’e behavior. While good 
training scenarios and good research scenarios have much in 
common, research needs are more rigorous. Frequently LOS 
research requires higher fidelity. For example: 

A central issue in simulation, whether it be used 
for research or training applications, is that of 
fidelity. For training applications, the require- 
ments for fidelity are straightforward in con- 
cept...; a high degree of fidelity is only useful if 
rt provides an effective training environment. 

iere is no a priori requirement for a given 
degree of fidelity, only for that which will pro- 

be U ne C fit ^ ^ l0 " g laStin S trainin 8 

For simulators used for research, the require- 
ments are somewhat different....Here th e a priori 
requirements for fidelity are more stringent 
because, by nature, research is used to explore 
the unknown.... (Nagel and Shiner, 1983) 


The first steps in any applied research are problem defini 
on sfudy plan develop,,, e„,. selccllon E ™m»l 

and ,h V 'n° p,eparalion - The choice of research vehicle 
and the decision of whether or not to use LOS methods are 
inherent parts of this early process. It is worth repeating that 

complicate 'tesL’y ZeZi'Jv 'and “ 7"^ "" b ° ,h 
cult in frill II y nneLessanl y , and make it more diffi- 
* 1 cntlca,1 y examine a narrow research issue 

i„ ZLlhT' reqUi,es "">** real-world fidel- 

I S , TT°y “" d limiBd ' bm h « h| y specialized 
yperlise. Olher kinds of parl-lask research might require a 

...odes, scenario sorts a fundamental deciZl 

cess o, a subsystem’s operability o, controllability, but hule 

Kraf T ‘ f ' aXkpi ‘- ” • 0. motion system 

h Tl dis 7" ed f basic problem ■” < app '" b « c> 

At the other end of the scale, research involving inter 
action of multiple subsystems that arc parts of the aviation 

skilled part, ana'? ^'“7 ^ Carefully scripted scenarios, 
and the Vull ? S ’ * n ’ ultldlscl P Iln ary investigative team 
1979) hC reS ° UrCeS t0day ’ S SlmuIators (Ruffell-Smith, 

All civil aviation systems include a man-machine system 

them rr f Craft ’ 3nd 3n ATC system which con ok 
them. While each is a complete and complex system in Z 

^?to^l t i mU ?-h be V ' eWed 35 3 C0 ° rd]nated 0 P^ation in 
, . y ' ^ ls re Port is concerned with the man 
machine system, which operates the aircraft, and its interface 
with a realistic simulation of the ATC system For our pur 

viewpoint 'we 1113110 " ^ r6alis,ic ° n,y cockpit 

p We are not concerned with simulation of the 
internal operation of the ATC system. ” 


Research scenarios also require greater control of crew 
performance. Deviations from the expected oulcoml of I 

Sim"] ^ H SCenan ° Can provide a Suable learning experience 
Similar deviations from a research scenario, particular when 

rrr y de,i r ns f, °™ the ^ 

behavior being studied, provide unwanted variance 

caZmqui ~T“ Ca " b “' b C0nf0 “ nd ™ W’ “d. in some 
cases, require a larger number of trials. 

A f naI difPerence is that when pilots are being trained 

e7Tec;£ri } 7: ADVANTAGES ° F UNMRIENraD FMS RESEARCH 

be m„ re tolerant of fidelit $]i in trajn y p * 0 Lme-onented simulation provides a relatively economical 

Th,s ,s particularly true if any aspect of thLsear^rc S n ^°“ T"?™ ^ ™ 

within the aviation system. It permits observation of the 
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system’s elements (hardware, software, : ''^ZSc The 
ment) under conditions that allow reasonable control of he 
variables in each element. Without control of those vanables 
it is difficult to identify the factors responsible for observ 

prmXal strength of line-oriented FMS research is its 
unique ability to study the subtle interactions involved in 
such areas as crew coordination, vlgilince • 
resource management 3 LOS can be required if the researcl 
involves conditions such as performance degradation, long 
duraUon or infrequent events, or response to emergencies or 
irregularities as a function of time on-duty. In studies of p i 
formance degradation, as might occur in some fatigue states 
the desired subject-state often can be achieved conveniently 
outside the simulator before the simulated m > sslon be ^n ^ 
The conditioned performance must then be evaluated 

fU ' Mother^ntage of LOS research is that it permits eval- 
uation of the performance of people and equipm 
occurs during mnsiiions from one flight plvt* >« 
mode to another. In some cases, the simulation of a comptet 
flight may not be necessary to accomplish the research objec- 
tive U is necessary, however, to faithfully simulate all of the 
flight phases or operational modes being examined and to 
simulate adjacent phases or modes to ensure that a y 
action effects that occur can be examined. 

LOS also can be used to study a related series of problems 
involving hardware, software, and behavior in which there are 
cleanbreaks be, ween ,he individual dements being evalu- 
a ed An example is a sene, of related par. -.ask studies 
mvlmg .be evaluation of specific insuuinenis. displays, 
onerational procedures, or controls (appendix L ). 

P When newly developed hardware or software are in r - 
duced into the system, its teal-world performance can be 
quite different from the performance that ”’ s * 

1 creators, or was observed during part-task studies during 
development. Performance in real flight, however, is re u 
mace criteria - for that is where the performance of hard- 
wa c software, and liveware have eri.ical importance Today 
LOS provides the best available ..hide fo, predie, mg real- 

is useful fo, studying individual - 
group behavior when the study addresses the follow, ng kinds 

° f rVdidarron of .he results of smaller studies or behav- 
ioral hypotheses by observing behavior in a total sys 
context. 


2 Evaluation of new or modified hardware, software, or 
procedures in a systems eon, ex. before fhey are nmoduced 

into the aviation system. f 

3 Problem exploration, such as the identification of sys- 
tem problems that occur when individual subsystems arc 

omb d in" the total system, or observation of the pertor- 
mance' 1 and interacts of individual and other element 
within the aviation system under the wide range of operating 
conditions to which they are exposed. 


“ 3 “Cockpit resource management refers to the utilization of all 
information equipment, and people U 
available resources - . n b er 1981 Murphy 

eiently utilizes available resources in attaining system objeU.vcs, 


COMPOSITION OF THE RESEARCH TEAM 

Once the operational problem and research question have 

been determined and a decision has been made Oo use LOS 
the researcher's next task is to select a research team. The 
team's principal tasks will be to develop the scenario and to 
run the experiment. The following sections discuss 
variety of skills and knowledge that will be require . 


developing the scenario 

The research problem always drives the scenario require- 
ments . This is because the scenario must produce behav or of 
interest to the researcher and the ultimate user Because 
resea ch must be done in a simulated aircraft and airline envi- 
ron, vent the first needs are for expertise in the aircra an 
type of operation that will be simulated. Long-haul short- 
haul charter, air taxi, and commuter are examples of type of 
operation Equally important is expertise in the geography 
area involved and in local meteorological and environment^ 
pheromena such as ice, snow and the slippery runways of 
winter operations, thunderstorms, valley fogs wind shears, 
and the special problems associated with high-altitude flying. 
Au traffic control plays a major role in an carrier operation 
and a high level of familiarity with ATC operations, inclu g 
Us vernacular and jargon, is another important requirement 
for it least one member of the research team. 

Meteorological exjrertise is stressed because weather is a 
inevitable part of LOS scenarios. The simulated weather mus 
be consistent with the weather encountered in the season a 
* ° leal area selected. Most areas of .he country have 
foo d weather characteristics which have considerable opera- 
Unal significance. Understandably, pilots can be acutely 
aw ire of and sensitive to, them. Considerably more than 
general familiarity with aviation weather phenomena 
needed to be sure that scenario fidelity is not degraded 
because of obviously implausible simulated weather 

Even the best simulators have limitations and specific 
knowledge of those limitations is required. It the researcher 
,s not intimately familiar with the simulator to be used and 
its current status, a person with that knowledge will be 

needed. 


Finally, a scrupulously detailed script has to be written 

™:V r ;? StantiaI advantages ln having a team member 
ho has had previous experience in writing full-mission sce- 

nuno scripts. If this is not possible, having at least reasonable 
Jccess to a person who has written successful simulation 
scenario scripts can be most helpful. 


running the experiment 


Much of the knowledge required to develop a good LOS 
scenario is also required to run one. Again, it is of prime 
importance to have people who are familiar with the opera- 
ion eing simulated and its ancillary services. Ancillary ser- 
vices can include cargo, maintenance, weight and balance 
ramp service, passenger service, fueling, dispatch, flight oper- 
a ions, and any others that might be involved in the simu- 
a cd operation. Pilot behavior will be an important part of 
virtually all studies, and a pilot familiar with the aircraft and 
its operation will be required. He or she will be needed to 
eal icalistically with the host of minor and largely unfore- 
seen operational problems that will occur, and to provide the 
icseaicher with a pilot’s perspective regarding them 

A professional air traffic controller, or someone with oral 
and operational skills very close to one, is a requirement for 
any scenarios that involve more than an absolute minimum 
off ATC interactions. This is particularly important because 

scrim ^d ATP Ullp0SS, . bll,ty of a]wa y s closely following 
scripted ATC communications when one side of the com 

mumcations loop is entirely unscripted. Of course, there is 

rMlkt? ii° SCUP i t ,hC P ' l0t S ‘ dC ° f the commu nications, and 
realistically simulate a real-world operation. 

The objection to the presence of observers in the simula- 
nt ta ' S Ti Ut fCW real ' worId airline operations have cockpit 
servers. There is simply no way that the presence of even * 
ent and unobstrusive observers can enhance realism. How- 
ever. a very practical reason for the presence of at least one 
non-crew member in the cab is that in all but the most 

ii ts :ir "" ,dcis - " ,c simuia, ° r ' s op " a,in * 

There are other advantages to having cab observers. They 
can iccord important behavioral and performance data that 
lS ° thc ™ di «icul* ^ obtain, and can monitor both the 

«cnZ a r h°V he SimUlUt0r and thC 8eneral Pr0 « ress oft]le 

scenario Cab observers can be particularly helpful when the 
reasons or unexpected pilot reactions or deviations from the 
scenario are no. entirely clear to team members observing 
from a remote location. Such conditions are bound to occur 
The cab observer, are therefore in an optimum position to 

nt> the situation by communicating directly with the 
team members outside. 

,-, Cab ‘| bservcrs < and simulator operators) have also served 
effectively as scenario directors in both training and research 
simulations when the simulated operational situation 


cases th h CrCW mei t0 leaVe the COck P il - In these 
cases the observer can control the time when the crew 

member “returns to the cockpit” and can give the returning 

crew member an appropriate operational message regard J 

nario m Th Ca . h “ eXtema ' CO,,dltions re qmred by the sce- 
nario. This technique assures that the pilot receives an opera- 

nal message from the returning crew member that facili- 
tates the scripted scenario. 

simuL h , er imPOrtam memberS ° f the 0 P erat ' n 8 team are a 
imulator operator and a simulator-maintenance technician 

The maintenance technician is required because occasional 
simulator malfunctions are inevitable. Many malfunctions 
an be repaired rapidly, but this takes specialized expertise 
If this expertise is not available, frustrating and expensive 
delays can occur. Such delays can mean the loss of the entire 
simulator exercise and, for purposes of the experiment the 
loss of a trained and carefully selected flight crew. 

Complete and accurate data collection is essential 
Depending on the nature of the study, designated individuals 
may be required for specific data collecting tasks. (Data col- 
c ion was discussed in some detail in chapter 2.) If such 
people are required, it is important not to burden them with 
other tasks which can interfere with their primary responsi- 
i hty. Even automatic data collecting devices need monitor- 
ing. Unfortunately, critical data lost is lost forever 

Finally, there has to be a “wagon boss” - an individual 
who is usually but not always, the principal investigator - 
lose job at this point is simply to coordinate and run the 
experiment. Prior LOS experience is an obvious help A LOS 

Id T v Undertaking because ^gardless of the planning 
and the preliminary testing, the flight crew may take actions 

which re unanticipated, the simulator may fail partially or 
completely, or the research team may make a mistake in 
controlling the scenario. Any of these problems will requ.re 
real- mie decisions that will affect the outcome of the study 
and the validity of the results. Planning, training, and leader 

tean^ 6 ‘° deVd ° P 3 wel, - coord mated operating 

shmf SU T ry ’ the rCSearCh t6am Wil1 need P° sitive leader- 
sh p from the principal investigator and individuals with the 
following kinds of skills or experience: 

1 . For developing the scenario: 

a- An expert in the type of operation simulated and its 
ancillary services 

b. An aircraft specialist 

c. An ATC expert 

d. A weather expert 

e. A scenario script writer 
2. For running the experiment: 

a ' , A t pereon . fa,niliar with the operation being simu- 
lated and its ancillary services 

c. W " h ,he “ d “ S 0!Xmi0n 

d. An observer (researcher) in the simulator cockpit 

e. A simulator operator and technician 
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f. An air traffic controller(s) 
o A “wagon boss’ 

The preceding discussion does not imply that a separa e 
people who have not had similar experience. 

research subjects 


Subject Selection 


Maior U S. air carrier pilots are ideal subjects for LOS 
J -h Thev arc familiar with the civil aviation system, an 

IT! pi! wich little or no previous simulator experience 
d if the research goal is to produce the equivalen o r 

*• w- * 

experiment training to ensure a high 

" C TCTiarchc, must always remember that pilot po, ro- 
mance will be influenced markedly by the operational prac 
“ of tie airline that employs them. Although there a, 
nearly universal principles of good operating practice, ther 
are also substantial differences in airline policies and proce- 
ss These differences include critical items such as 
required call outs, the assignment of duties, a somet^c 
undefined expectation of what to expect from other 

performance of regular line crews Is a research 
requirement, there are significant advantages in us gfc 

Ki ^^r:^ru^t::es»m«».r.. ? 

me used o, an entire study, the da, a collected may be rep e- 


arc not substantial. . * *i,p 

Performance can be influenced by the type ot tups the 
Perloimance c short-haul operations 

*• resu ! is ““ ld , b jst“ 

different from the same studies conducted with pdote 

Mv -tnuliar with the intricacies and pace of short-haul 
dons The converse is equally true. Wide variations m 
plot familiarization with the operation being 
provide unwanted variability that can be difficult g 

Designer s' of LOFT scenarios consider these differences, 
but e v them from a training rather than from a researc 
no nt cf view Their task is to develop scenarios that deal 
with the operational problems of a specific pilot popula 10 
aircraft, procedures, and route structure. As noted by Laubcr 
and Foushee (1981): 


The design and development of scenarios for 
1 OFT programs requires considerable at ten ton 
!f the needs of the particular earner. Ditierent 
air carriers, different operations within a earner 
and different pilots within an operation all have 
various types of training needs. U « 
that considerable flexibility be permitted m 
order to meet these various training 
requirements. 

Similarly, the design and development ot scenarios for 
resea ch will require considerable attention t° resear^i o j 
ives and to the impact of subject pilot experience^ ffc ng 
air carriers, differing operations within a earner, and d tc g 
pilots within an operation can bring a variety ° ^ ' ' 
snccuvcs and behavior into the experiment. It ls essC " t 
Z ' these factors be recognized in the selection of the pilots 

training and cliecking sessions include procedures, ca cula- 
started If the research objective is to study typ.cal P«*w. 


19 


A basic principle in LOS is to minimize the number of 

Sz s „u„' r n , d t cr,!ws ,im ' " ,ey are 

and not in a routine line operation. When pilots are asked to 

e altd° tofif 8 !! Well ' eStabIis,ied operating procedures, or 

dures and " T * membCrS Wh ° USe dissimiJ ^ Proce- 
dures and callouts, it can only serve to periodically remind 

them that they are not involved in a real-world operation 

The simulator should be configured as closely as possible 

0 the equipment the subject pilots are currently flying If t 

needed ^ SuTT' ^ ‘ WfCrenCCS “W 4 be 
’ f. h tralmn 8 is always undesirable, but may be 
unavoidable. many caaea changing critical nigh, “a,, m 

t " . (S , Uch . as fllght directors w course or attitude indica- 
ors) to duplicate a specific airline cockpit configuration can 
produce more realistic and, therefore, better performance 
data than the best differences training. Changing the simu 
at°r configuration might be less expensive than the time and 
materials required for differences training even if the flight 
crews are carefully selected and scheduled. 

„,.I hcrC is another difficulty with training for substantial 

1 erences. Although pilot performance may appear to 
be adequate under benign training conditions, the basic ten 
dency in all individuals to revert to old and well-established 

bl J f Pat . terns under hi 8 h workload conditions or stress levels 
unf lf l T 1 t0 ° VerCOme ' Pll ° ,S in 3,1 U " famd - -C using 

trvine to" 1 " ™ y Sp<?nd time (and nlental capacity) 

> ng to remember how to do something rather than concen- 

1 8 ° n what to do - Suc b reactions during a LOS research 

ZZT: dependins on the study ' s <****«. p^oe 

ss than optimum (or even misleading) performance data 
Manuals and forms are equally important. There are wide 

forms °" S m the design of operating manuals, opemtioial 

turers' * “ Printed materiaI amoil S airlines, manufac- 
turers, and corporate operators. Familiar software of this sort 

adds realism. Unfamiliar software, especially if it will be used 

unfm S ' ,Uatl0nS ' de 8 ra des realism. More importantly 
unfamiliar software can also be an unrealistic source of com 

f " m W ° uld not be representative of operations in a 
faimlnir environment. It can degrade information-seeking and 
possibly subsequent decision performance. 8 

classic^Drohlp 16 ^' ** “ VlrtUal,y im possible to avoid the 

tee sub ect h' aSS ° C ' ated ^ depCndc » ce on volun- 
teer subjects. However, most seasoned researchers who have 

worked with airline pilots believe this is a relatively minor 

Pr em, particularly if the experiment is planned with a 

generous number of subjects. * n 


Subject Training 

the S ;r tralmn8 needs ' sub J ect training procedures, and 
he entenon for training completion need to be established 

in the JT been dete rmined. Each was discussed 

He! t 1 CUSSi ° n ° f SUbject trainin § in chapter 2 

simulator^ 3 r t,0na ' ° f ° perational currency and 

simulator familiarization will be considered. 


The importance of operational currency is reflected i„ 

malion there is little question that pilots who Ze „« 
usuTr standard! P '' ,0llS d ° n °' ■» their 

" "Prions are being studied, research subiects 

ated because pilots cannot be expected to display routine 

XenT oT'fmn ,n “ ‘ f d ~ « ZZ Z 

“ ' "I of full hne-ortented familiarization. On the other 
hand, narve subjects - or experienced pilots who are „„,Tu" 
can be acceptable or even desired in some instances 
” a , T” “ ,ha ‘ identifies the 

chaTacierisZ “ SUbJeC ' S «“ d ad 

do S’!!' ‘lT' SSl 7 advances, simulators still 

realltsue I T '" <»»< very 

) sue recognized in appendix H of Part 121 of the 

Federal Air Regulations, which requires recency o exp - 
ence m simulators if they are to be used in nilJ t • • P 
checking. Part 121, appendix H.also provides specif* 

There 7 r " S '* b,,sh,r * simulator currency if it has been lost 
There is an even greater need for simulator currency if the 
simuhtor will be used for line-oriented research 

Currency m both piloting skills and in the equipment 
being simulated is needed because of the complexity oHhe 
flying task and the individual characteristics of modern tram 
po^atreraff One sesnon of differences Paining can o Z 
a DC-9 pilot into a B-737 pilot with equivalent skill The 
necessary degree of familiarization with the simu a, or Ld 
= manuals, forms, and trip p.petwork required ° 
study can vary, but should be sufficient to minimize learnimt 
behavtor once the expenmen, is under way unless JeaS 
behavtor ts an objective of, or will not unduly confound, the 

derW S ed fU bv 8U et anCe 0 " ^ am ° Unt ° f training needed can be 
dertved by examining the transition training syllabus of a 

2;r:: a d tiv T e he airline th : type of ope -ion a nd : q ° u ; 

rnem used. The researcher, however, should recognize that 
hese are minimum requirements and that in only a few cases 

they produce the equivalent of an experienced pilot 
operating in familiar conditions. P 

The timing and coordination of cockpit procedures is a 
particularly important component of flight crew operations 
The required skill appears to decay rapidly with disuse and 
minor procedural hesitations can have an adverse effect on an 
lerwtse smooth and professional performance. This can be 


20 



an important consideration in LOS research that is 
dependent on good crew interaction and coordination, espe- 
cially if the research includes high workload periods 
Recency and currency may not be an issue w en 
hardware, software, or procedures are being studie . n e 
cases a degree of learning behavior is inevitable, as it is dur 
ing the “shakedown period" that occurs when new hardware 
or software is introduced into a line operation. Some training 
and familiarization will be needed. The scope of such training 
will depend on the research objectives, experimental strategy, 

and Aldiough periodic testing is a large part of a professional 
pilot's life, very few pilots enjoy it. and, regardless of profes- 
sions to the contrary', pilot subjects are bound to perceive 
an element of testing in any situation that requires a demon- 
stration of their professional skill. Therefore it is essentia 
that high-quality training be given, and that high standard of 
confidentiality and anonymity be maintained. It is cqu y 
important that these factors are perceived as such by the 
pilot participants. Subject egos are important. 


A Basic Limitation 

It should be recognized at the outset that full simulation 
of an airline environment is simply not possible. There is no 
way that an airline flight crew can be expected to drive out 
to a research institution or training center, climb into a larg 
box-like room which is supported by intricately configured 
hydraulic cylinders that are surrounded by masses of e cc- 
tronic cables, and not be acutely aware of the tact that they 
are not about to fly a routine passenger flight. 

Fortunately, most airline flight crews are familiar with 
simulators and have learned to “play the simulator game_ 
They can be expected to become very much involved in the 
simulator exercise. If they are given a well-designed scenano 
they also can be expected to make a good faith effort 
react in the same way they would if they were faced w h 
similar stimuli under real-world conditions because this is the 
way that most of them have been trained and are routinely 

checker . 


THE SCENARIO 

“All LOFT scenarios and flight segments should be 
designed on the basis of a formal and detailed statement of 
specific objectives and desired end products (Lauber and 
Foushee 1981). This principle is even more importai 
LOS research because of the number of performance options 
which can arise from a realistic LOS scenario. Performance 
options are not a comparable problem in training because 
they can still have a significant training potential At worst, if 
the pilot docs not follow the expected procedure, 1 ca 
result simply in a need to repeat the training exercise. , 
however, undesired performance options are exercised 
research, it will complicate analysis by adding unwanted and 
confounding performance. 

The scenario can be developed as soon as the resear 
objectives have been defined. Scenario development is a sur- 
prisingly lone and painstaking process which can take consid- 
erably more time than it does to actually run the experiment. 
Subject matter experts in the airline operation and the air- 
craft being simulated are required. Also, SMEs in areas such 
as local meteorology, dispatch, passenger handling and main- 
tenance policies and procedures should be consulted if 
areas are or could become, a part of the scenario. 

Details are critical. For example, the weather situation 
should be consistent with real-world weather patterns that 
normally occur in that geographical area at the tune of ic 
simulation. Pilots will recognize it if the weather is not realis- 
tic and judge the scenario accordingly. When elements that 
will satisfy the basic research objectives have been deter- 
mined, a timc-event-linc description of the operational tasks 
that arc required should be one of the first assignments. 


Elements of Successful Research Scenarios 

Successful research scenarios have included such items as. 

1 Sufficient workload to discriminate measurab e varia 
bles in the performance of interest. (Cockpit workload is a 
complex and difficult subject which has been detined or con- 
sidered in a great many ways. It includes decision making 
time-sharing and prioritizing concepts: physical and mental 
tasks- and the control of a wide variety ol system-relevant, 
but not always operationally critical, considerations.) 

o Sufficient time to permit meaningful decision pro- 
cesses and crew interactions. (It is a considerable oversimpli- 
fication to note that there are at least two kinds ot opera- 
tional decisions. In the first type, there usually is sufficient 
time to consider the available operational variables. In the 
second type, critical operational decisions must be made very 
rapidly, with little, if any, time for evaluation Scenario 
designers should be sensitive to the importance of the time- 
available variable in the decision process.) 

3. Assurance that fuel available is a meaningful factor by 
careful selection of weather, route, and payload. (Varying 
the time of preplanned holding periods is one cffet ive 
method of controlling the “remaining fuel variable. It 
worth noting that fuel management has become an increas- 
ingly important consideration in contemporary cost- 
conscious airline operations.) 

4. Use of both scenario events and the environment as 
driving factors in the scenario. (Deteriorating weather, icing 
thunderstorms, cross- or tailwinds, wind shear, and wet and 
slippery runways are examples of environmental elements 
that have been used successfully for this purpose.) 

S Provision for a number of decision choice points, 
including the provision of some during flight planning. (Pro- 
viding decision points during flight planning is a good way to 
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get the night crew involved in the simulated action at an 
early stage. Decisions made during night planning regarding 
items such as fuel, minimum equipment list (MEL) items 
and other inoperative equipment can affect events and deci- 
sion making for the entire night and often will determine the 

I t S h ele f, 0 " ° f °P era tional anomalies that will best pro- 
duce the behavior of interest. (This clearly requires opera- 
. , ^ x P ertIse ln all aspects of the operation being simu- 
a e . t le operational experts are to be maximally useful 
he researcher must be sure that the experts have a good 
understanding of the research objectives.) 

7. Provision of more than one viable course of action for 
he crew to facilitate emergent behavior, rather than behavior 

Of th’e 17 ,7 "I*" ^ thC C ° nStraintS ° f the “cnario. 

ell 1 1'"' d£S observati,)n ^ flight-crew decision pro- 
the s | r « e archer is interested in emergent behaWor 

the scenario should include reasonable operational options 

have er b W e 1Se 7 ° nly b6haVi0r tMat WHI CmergC Wil1 aheady 
have been determined by the logical constraints of the see 

f n0 . . inversely, if the researcher is interested only in a spe- 
ll' l I! 10 "’ the " Umber of reasonable operational 
variables should be restricted. To do this without destroying 

s:7ir sm rcquires co,,s,derabie ski " " 

8. Inclusion of cabin crew and other ancillary services 

tTre'cabI ^ (,n ma "y ^al-night situations, 

the cabin crew and ancillary services are, or should be, 

mvo vc m some cases only by being kept advised of the 
progress or events of the night. Appropriate flight crew/cabin 
crew d.a ogue adds a great deal of realism to the simulation ) 

9. Selection of performance requirements that arc within 
e ordinary skills of individual pilots and within the skills 
of an integrated coordinated crew. (The capability of modern 

‘ »*'y of operational i,Z 

" “ " r nd , Cn, " Ee " aK ««<• > temptation fo, the 

caT„ T ,i C °”' P ,h ' Sl " , “ hlCd "P'eatJcn beyond 

reason. This can cause resentment and is an almost certain 

way to destroy the night crew’s “illusion of reality.’’) 


diagnosed and properly handled, the engine can be restarted 
and considered a normal engine for the remainder of the 
ig i - At the other end of the complexity scale, malfunction- 
ing landing- or training-edge flaps, or the loss of one or more 
~ if 21 -- bu -’ - hydraulic systems can aZ 
aTtlie gate ^ ^ the is parked 

Within these extremes, there arc many variations and 

example" 1 ''! 65 7 “ h ° St °' lmaginative modifications. For 
xample, a loss of pressurization at high altitude can cause an 

emergency descent which in turn can produce a period of 
ugh workload and serious internal and external communica- 
tions problems. Once the lower altitude is reached, however 
the airplane becomes essentially normal except for the pres- 
sure loss and a significant increase in fuel consumptioii In 
ost cases this will require alteration of the night plan and a 
landing short of the destination or planned alternate It can 

P ln ZIT t0 minimiZe thc numbcr of available 

ln one imaginative LOFT scenario, a simulated bomb 
xplosion m a baggage compartment was used to considera- 
comp lcate a loss-of-pressurization problem. Thc simu- 
lated explosion resulted in the loss of pressurization. How- 
ver, the explosion also caused aircraft subsystem problems 
and unverifiablc structural damage. Cabin crew, ATC dis- 
patch and emergency services were all involved in the 
scenario. lI,t 


Real-World Performance 


The Operational Problem 

There are virtually no limits to the kinds of operational 
problems that can be simulated. Problems in hardware soft- 

saud'ied'TT' e " Vlr0nmcnt ’ and th eir interactions can all be 
studied. Each category can originate from within thc cockpit 

or outside of it. Operational problems can range from rela- 
tively simple problems which have no further impact on the 

Plex probL y 7h diagnosed and corrected, to com- 
plex problems which cannot be corrected in night and have 
continuing operational ramifications. 

nr ^ g ' ne A S f tarting P r «blems are a good example of a simple 
1 blem. After a “hung,” or a potential “hot” start has been 


It is sometimes difficult for the researcher to evaluate 
pilot behavior in terms of its real-world significance if the 
vcl of performance is less than expected, or when there has 

7nda a blv 7i Crmr ’ inClUding Un Crr ° r in j udgmc ”t- Under- 
s andably, pilots are sensitive about their professional perfor- 
mance. A frequent response, and it can be simply a normal 
defense mechanism, is to say: “Of course I knew we were 7 
a simu ator If it had been an actual line operation I would 

777 T diffe ? tly ” 11 Wiil bc virtual 'y Impossible 
for the r searcher to know the truth of such statements 

Occasionally, some extraordinary behaviors have been 
involved in air transport accidents. 

Evaluating the real-world equivalent performance of sub- 
ordinate crew members or of the monitoring pilot (the pilot 
“ "*!«>. k P-Ucn,. Piles can also be s c„Z t 

fle P mavT IU "! Crew m0mbcrs - They, Ihcrc 

ye more reluctant to question judgments or to 

Point out errors in a simulated environment than they would 

7 rClatlVC privac y of an airline cockpit. The current 

thc7eed 7 t0tal performance - including recognition of 
he need for resource management and incapacitation train- 

mg (which, among other things, stresses the importance of 
mon.tor.ng and full participation of all crew members) helps 
minimize this problem in line flight operations. A specific 


reference to the importance of these operational concepts 
during prcflight briefing (in the context of wanting all 
aspects of real-world behavior) can help mitigate a reluctanc 
to fully participate during the simulator exercise. 

Despite these inherent difficulties, there is a clear consen- 
sus among line pilots, instructors, and researchers that some- 
thing very close to a total line environment can be • 

Even more important, they believe that in a realistic simula- 
tor exercise pilots become so engrossed m their operating 
problem that they respond as they would in real 1 igi • 
is little question that it is important tor the researcher to 
make all aspects of the simulated exercise as realistic as pos- 
sible and to avoid minor intermittent stimuli that jar ie 
pilots back into the world of the simulator with even small 
cues that are unrealistic. 

A Final Comment on the Importance of Scenario Realism 

As noted in Cody’s McDonncll-Douglas simulation studies 
(appendix 1), the main concern of pilot subjects is mission or 
scenario fidelity. Pilots do not readily accept deviations from 
operational practice unless the purpose ot the study is dearly 
to try out new equipment or procedures. Part of their con- 
cern with fidelity stems from the fact that a simulation 
(which is essentially a duplication of an actual mission) is a 
test of their own capabilities. If pilots are to submit to such 
testing they understandably want high levels ot fidelity o 
maximize their opportunity to perform properly. They do 
not want a shortcoming in the simulation to be interpreted 
as a lack of personal ability. 

THE SCENARIO SCRIPT 


Line-oriented flight training experience has demonstrated 
the importance of detailed scenario scripts. Creating the 
illusion of the real world requires great attention to dctai 
To an even greater extent than in training, t laviora 
researchers need maximum control of performance. It is 
virtually impossible to achieve an acceptable level of eontro 
using a aeneralized script. The problem is even greater it the 
researcher is also part of the scenario control team. The 
additional workload and concentration required by an invari- 
ably futile attempt to achieve a realistic scenario rom a 
generalized script leaves the researcher little time to observe 

the performance being studied. 

It is mandatory to script all communications and to use 
them verbatim. Airline mechanics, dispatchers cabin crew, 
passenger agents and other aviation personnel all have their 
particular communication styles. Virtually any ot them can 
he involved in realistic scenarios. Air traffic control commun- 
ications are most important. If it is at all possible, a working 
a,r traffic controller should be used to provide these 


communications. Even then, messages should be scripted 
meticulously to minimize spontaneous innovations S ponta- 
ncous innovations, while occasionally necessary, are almost 
alwavs jndesiruble. 

The script should specify the timing of all communica- 
tions and other elements of the scenario. Each event s 10U 
be placed on a time/event line which must be scrupulous y 
followed. The script should indicate probable crew responses 
as well as alternative responses to the extent that they can be 
predicted. Because the researcher can expect considerable 
variation in individual and crew performance it may be desir- 
able to script some kinds of simulated problems by aircraft 
status or position rather than chronological tune Examples 
rnieht be when the aircraft reaches a given luel state (see 
Murphy, appendix B) or is a specified number ot miles or 

minutes from a geographical fix. , 

A most difficult problem is to realistically control the 
number of options that are available to the flight crew with- 
out reducing them to the point that the researcher can have 
no confidence that the scenario is producing the equivalent 
of real-world emergent behavior. Even when emergent e rav- 
ior is not required or desired, it is essential that scenario co - 
trol devices be operationally realistic and tightly scnptecE 
Conti ol mechanisms that have been used successfully 
LOFT exercises include the following. 

1. Sudden weather deterioration below landing 

mi T‘ Passenger service considerations and in-flight passenger 

emergencies. . . 

3. Runways closed for maintenance, snow plowing, or 

disabled vehicles on the runway. 

4 Bomb threats, or hijack attempts. 

5 Subsystem status uncertainties or failures. 

(, Traffic delays. 

7 Obvious or subtle crew member incapacitations 
These and similar kinds of events can be effective. They also 

happen in real flight operations. 

scenario control devices should be used with considerable 
discretion. There is always the possibility that the real mes- 
sage the crew gets when control mechanisms arc used is no 
the scripted message, but the reality-destroying message ha 
in tins contrived and make-believe world, the researcher docs 
not want the crew to do something that they would have 
done m a line operation. If that happens, the n«vitab e rca^ 
tiou is "Well we’re back in the simulator again. 1 rom 
crew’s viewpoint, losing a viable alternate for an aircra t 
operational reason, such as the reduced range available 
because of an engine or pressurization loss, has much mor 
realism than a sudden “truck on the runway. Meteorologi- 
cally sound weather changes, including changes in win s 
aloft, have inherent plausibility because ot the uncertainty < 
precise weather forecasts. 
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SPECIFIC CONSIDERATIONS 


Route Selection and Scenario Reality 


Communications associated with ATC services are complex 
and will be discussed in the next section. 


The illusion ol reality is enhanced for the flight crews if 
the route selected for the experiment is one with which they 
are familiar for at least could encounter in their day-to-day 
ymg). Familiar intersections, radio navigation aids, and air- 
ways reinforce the validity of the simulation and help main- 
tain the illusion of a line operation. Reality is also enhanced 
t rough the use of realistic call signs, including airline names 
and appropriate flight numbers. Today, airline charter opera- 
tions are widespread enough to accommodate virtually any 
research scenario needs, as long as the route segments are 
chosen carefully and imaginatively. 

Most airline pilots are familiar with the routes they fly. 
Although there are variations among airlines, pilots normally 
have advance notice of their flights. This permits prior 
review ot approach and departure procedures, special terrain 
or other geographic considerations, likely routing, general 
weather patterns, and other relevant factors. If a routine line 
operation is desired, these are important preliminary consid- 
erations in the selection of the routes to be used and the 
Pilots who will fly them. Unless it is contra-indicated by the 
research objectives, the pilots should know the flight that 
has been planned for them. 

Weather, including turbulence, which is typical of the 
geographical area and the season, adds a great deal of realism 
to a line-oriented scenario. Conversely, simulated weather 
which is no. inherently plausible, and this includes all ele- 
ments of weather, significantly degrades it. 


ATC Communications 


Navigational Aids and Communications Services 

All of the navigational aids (NAVAIDS) that are normally 
on the selected route should be simulated faithfully This 
includes providing their proper identifications. If any 

AVAIDS will not be available, their absence should be 
stated in the Notice To Airmen (NOTAMS) which should be 
available as part of the preflight papers. Any radio-aid identi- 

J™"" S ' vh ' Ch Cann °‘ be simulated should be properly 
NOTAMed. Pilot use of NAVAIDS not required but nor- 
mally available on the route can be operationally sound (e.g. 
for double-checking position) and, for the pilot, is an indirect 
method of checking the validity of the simulation. 

Communications to at least three outside sources - com- 
pany, cabin crew, and ATC will be required. Company com- 
munications can involve dispatch, weight and balance, pas- 
senger service, maintenance, ramp service, cargo, fuel, gate 
information, and so forth. These communications vary con- 
siderably among airlines, and require careful scripting and 
Jmilianty with the operation being simulated. Cabin crew/ 
cockpit crew communications arc equally important. 


Todays airline operations involve an ATC communica- 
tions contact with a minimum of 11 different controller 
functions on each flight (e.g., clearance delivery, gate hold 
ground, tower, departure, low-altitude enroute, high-altitude 
enroute, low-altitude arrival, approach, final approach 
tower, and finally ground again at the destination). In many 
cases pilots will communicate with at least that many individ- 
ual controllers. It can be a nice touch if pilots do not hear 
the same voice performing each controller function. 

This by no means suggests that it would be feasible, or 
even desirable, to have 1 1 individuals for ATC communica- 
tions, but it does enhance realism if the same individual does 
not simulate all of them. Also, it reduces the possibility of 
potential, momentary confusion during a hand-off from one 
controller to the next. The first reaction of a pilot who hears 
ie same voice after changing frequencies to the next con- 
troUer might be to think that he forgot to change frequen- 
cies. This is particularly true in a period of high workload 
where several actions might be time-shared. Thus, if there are 
at least two people available to simulate ATC communica- 
tions they should alternate, so there is a voice change for 
each hand-off. To deal with this problem, the NASA Ames 
Man-Vehicle Systems Research Facility (MVSRF) utilizes an 
electronic voice disguiser that provides 12 different voices 

Irom a single controller to enhance the realism of its 
simulations. 

At some point, background materials such as ATC tapes 
must be secured, or appropriate scripts developed. They must 
be typical of the airways and terminals selected, the time of 
the simulated flight, and the simulated operation. An ATC 
tape giving visual approaches during simulated CAT II 
weather, conflicting wind information, or inappropriate 
clearances to other airplanes can destroy the realism of an 
o herwise effective scenario. It is important to be sure that 
all background communications are consistent with the oper- 
ation being simulated. 

There is little “open” ATC communication time during 
peak operations at busy airports such as ORD, LAX ATL 

DCA ’ or SF0 - Scripting and then simulating realistic 
IC communications at such airports during their peak traf- 
fic periods is very difficult. Foushec (see appendix A) has 
reported considerable success by using taped recordings of 
actual communications to provide a realistic ATC communi- 
cations background for busy airports. This is also an effective 
way to increase scenario realism with the introduction of 
additional ATC communication voices. 

There is a great tendency among pilots to short-cut ATC 
and other communications protocols in simulator operations. 
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It is well worth making a special effort to maintain proper 
communications procedures. 

Flight Planning, Dispatch, and Preflight 

A continuing task of the scenario manager is to create and 
maintain an illusion of reality. This requires meticulous 
attention to the smallest details. As in real flight operations, 
the scenario should start with flight planning and dispatch. 
Preflight duties, the cockpit setup, engine start, pushback, 
and taxi are equally important because these are the items 
that set the stage. All ot them should be carefully scripted. 


Interaction with Cabin Crew and Passengers 

Simulation of the interaction between the flight crew and 
its cabin crew and passengers should be a part of virtually all 
line-oriented scenarios. This is because two-way communica- 
tions with the cabin crew and public address (PA) announce- 
ments to the passengers are an important part ot routine 
operations and an integral part of many abnormal and emer- 
gency procedures. If these interactions are not simulated 
effectively, it breaks the flight crew’s “reality chain” any 
time these communications are appropriate. It is particularly 
important not to break the reality chain during critical 
portions of the scenario. 

While a basic limitation of modern simulators is that their 
motion systems require the cockpit to remain sealed through- 
out the flight, this limitation does not prevent effective simu- 
lation of flight crew interactions with the cabin crew and 
passengers. The effective simulation of these critical elements 
of good scenarios requires only that the interactions arc 
scripted carefully and imaginatively, and that an operative 
cockpit-cabin interphone and passenger address system are 
provided. 

The growing number of male flight attendants has made it 
possible to use either a male or female voice tor communica- 
tions from the passenger cabin. Operationally critical com- 
munications, such as those involving emergency evacuations, 
the whole gamut of cabin emergencies, or problem passenger 
behavior, can be scripted and add considerably to the realism 
of the scenario. 

If it is appropriate to send a cockpit crew member back to 
the cabin for a first-hand evaluation of a problem, even this 
can be simulated eftectively by requiring that crew member 
to get up and leave his or her seat. The apparent return of the 
crew member to the cockpit should be carefully controlled. 
The real-world operational effect of the cabin visit can be 
realistically substituted with a scripted briefing to the return- 
ing crew member from the LOS coordinator or other 
observer in the simulator cab. 

When this happens, the illusion of flight is preserved for 
the crew remaining in the cockpit. The other crew member is 


doing what he or she would be doing in real flight (i.e., get 
out of the seat, leave the cockpit, evaluate a situation, and 
report hack). The overt behavior is consistent with reality 
and is operationally relevant. Under these conditions, the 
cockpit workload is usually high, so the obvious physical 
inconsist encies may pass unnoticed by the remaining cockpit 
crew. II' the scripting has been done well, a positive impres- 
sion will also have been made on the crew member who left 
the cockpit. 


Pacing, Tempo, and Quiet Periods 

The oacing and tempo of scenario elements can play a 
large role in creating an illusion of actual line operations. 
While there are occasional high-workload periods, routine air- 
line flights arc generally low keyed and relaxed, Emergencies 
and abnormal situations do occur, but they are rare. It is 
important that the scenario designer create this general atmo- 
sphere i an airline environment is being studied. 

The tempo should be consistent with the operation being 
simulated. Periods of relative inactivity (or quiet periods) 
should be scheduled as they occur in the real world. Even ii 
this is explained in the preflight briefing, it will be impossible 
to eliminate the pilot’s strong suspicion that the research 
scenario will involve considerably more than just a routine 
flight from A to B. and that any quiet period is simply a pre- 
lude to an ingeniously contrived flight problem. Usually, of 
course, this will be true. 

Mosi airline training simulator sessions consist of two 2 -hr 
sessions and arc limited to a total of 4 hr. It will be difficult 
to maintain an illusion of reality for longer periods, particu- 
larly those that involve prolonged cruise segments. Long 
periods at cruise require little pilot activity. These periods 
can be boring in an airplane, and very boring in a simulator. 
It is particularly difficult to realistically simulate the cockpit 
environment of long distance nights. If it is a night flight, the 
problem is exacerbated. 

TRAINING OF THE OPERATING TEAM AND 
SCENARIO TESTING 

The amount of training and indoctrination required tor 
the operating team will depend upon the complexity of the 
experiment, the skills of the individual team members, and 
their LOS experience. In addition to being experts in their 
field, all team members should know the research objectives 
and the simulator’s strengths and limitations. They should 
have a general understanding of the airline operation being 
simulated and a detailed knowledge of the scenario and the 
script. 

Special training in flight operations, observer techniques, 
and the making of value judgments may be required it any of 
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the specialists are to be used as observers. Dual qualification 
of the operating team members which permits them to func- 
tion as backups for other team members is helpful because 
schedule conflicts and availability issues can be expected 
during the course of any reasonably long study. 

Regardless of the care with which a research scenario has 
been designed and scripted, it is mandatory to plan a series 
of test runs or “shakedown flights” to ensure that: 

1. The scenario produces the desired test situations, 
Performance measuring devices produce the required 

data, 

3. Recorders record, 

4. Microphones transmit without feedback, 

5. Ail of the myriad details involved are polished and 
fine-tuned, and 

6. The scenario cast demonstrates that they can perform 
their assigned roles smoothly and realistically. 

This is the final step in LOS scenario development and it 
is critically important; it will take considerably more time 
than expected. Any but the simplest scenarios will require 
several iterations, and the researcher should not be satisfied 
until the entire scenario has been run without any “hitches.” 

PREFLIGHT BRIEFING FOR PARTICIPATING CREWS 


After the scenario has been fine-tuned and the subjects 
have been selected, a preflight briefing by the researcher is 
needed to ensure that the pilot subjects understand their role 
and the purpose of any special training that may be required. 
This is an ideal time to furnish general operational details 
such as the flight origin and destination, copies of typical 
flight plans, w-eight manifests, and loading forms. 

It is important to give subjects only general information 
that will not reveal parts of the scenario. For example, if a 
flight that normally takes 1 hr is planned, but the pilots 
have been told to plan for 2 hr in the simulator, there is 
obviously additional time to be accounted for. The alter- 
nates available under these conditions, with or without an 
ATC hold, will be apparent immediately to flight crews 
familiar with the geographical area. In addition, their behav- 
ior in the simulator can be influenced by their own specula- 
tive assumptions regarding the reasons for the inclusion of 
the additional time. 

Pilots should fully understand the “game plan.” Inade- 
quate briefings have created problems in LOFT, and can 
create greater problems for LOS research. Unless contra- 
indicated by the research objectives, some familiarization 
with the study objectives is desirable. The crew will certainly 
know that they arc involved in some sort of research. Not 
only will a briefing of objectives help them bridge the gap 
between the real and the simulated world, but if they are 
left in the dark, at least some of them will try to deduce the 
desired behavior, and modify their normal performance. This 


point is crucial and should be stressed when routine line 
behavior is desired. Even with an ideal prestudy briefing, it is 
difficult to avoid a certain amount of “Hawthorne Effect.” 
An unfortunate by-product of subject anonymity or con- 
fidentiality, which is a requirement for most research proj- 
ects. is a diminished personal identification with the outcome 
of the simulator exercise. Fortunately, this may be a minor 
consideration. Pilot egos are strong. Under nearly all condi- 
tions, they will try to produce a professional performance. In 
the preflight briefing it is important to stress the point that 
the pilots were selected because they are professionals, and 
that the research is dependent upon their professional perfor- 
mance. The research studies that are probably most sensitive 
to this issue are those that can produce degraded perfor- 
mance (e.g., because of severe fatigue, or for any other 
reason). 

It is critically important to point out that all supporting 
aspects of a regular line flight will be available. There arc sub- 
stantial differences among airlines, and the only reliable 
source regarding a specific airline s procedures is someone 
who knows that airline well. 

Once the simulation becomes airborne, the same rules 
apply. Full company radio facilities should be available at all 
times. Pilots not Hying should perform their functions 
exactly as they do on the line. Required operational report 
forms, including log book and emergency or irregularity 
forms, should be provided and used as is appropriate. 

The crew should be asked to role-play exactly as if they 
were on a regular line flight. It should be stressed that if any 
events which are not a part of the scenario (including simula- 
tor malfunctions) occur, the Hight crew will be informed 
immediately. If this is not done, there is always the possibil- 
ity that the Hight crew may mistake a scenario-induced prob- 
lem for a simulator malfunction. It also can help save a 
research run in which a simulator malfunction does occur. 


RUNNING THE SCENARIO 


The schedule must allow plenty of time to get started 
because there can be many last minute details that require 
attention by the researcher or the flight crew. As in LOFT, 
there should be no interruptions of the scenario once an FMS 
has begun. There should be no observer or researcher inter- 
lace with the flight crew other than in a simulated crew 
member exit from the cockpit, or in a simulated visit from a 
cabin crew member. 

The only exception is the case where the scenario must be 
interrupted to change simulator configuration or collect data 
that can be gathered in no other way. In these cases, one has 
to create plausible events (see appendix B for a visual system 
changeover), or interrupt the scenario at natural break 
points. We emphasize, however, that scenario interruptions 
should be made only as a last resort and that the researcher 
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will have to take special precautions to avoid any “cascad- 
ing” effects (appendix E). 


Deviations from the Scenario 

There is a high probability that there will be deviations 
from the scenario, and that the scenario operating team will 
have to cope with them. Deviations can come from straight- 
forward operational decisions, such as the time when deci- 
sions are made to divert to an alternate or the amount ot 
time a crew is willing to hold with a given amount of fuel. 
Many deviations are predictable, and should be included in 
the scenario as scripted alternatives, in some cases, subscc- 
narios may be needed to get the flight back on the track. 

Captains always have a final “emergency authority ” 
which permits them to take any action which in their judg- 
ment is necessary to preserve the safety of flight. This can 
include actions such as landing below minimums, proceeding 
without or refusing an ATC clearance, and diversion to an 
unauthorized airport. This authority is not used often in real- 
world flight operations both because “emergencies are rare 
and also because most pilots have an antipathy to writing 
reports of any sort - especially reports which automatically 
trigger an official investigation. 

The researcher should be aware of the captain’s emer- 
gency authority and should know that it can and has been 
used in simulator exercises. Here traditionally, there has not 
been (but in research there should probably be) a 1 equip- 
ment to complete an emergency report after landing and 
defend the action taken. Regardless of the research protocol, 
the area of “declared emergencies” (whether realistic or not) 
can be one of the most difficult areas in which to achieve the 
equivalent of real-world performance. 


Unexpected Poor Performance 

One rare occasions, an obviously poor performance, which 
can include classically poor judgment or even simply poor 
role-playing, can create a problem for the researcher. Fail- 
ures do occur during the routine training and checking of 
experienced crew members in regular airline operations, and 
although the failure rate is very low, one has to be prepared 
for this possibility in LOS research. 

The data secured in any instance of unexpectedly poor 
performance may or may not be useful. Although this is a 
judgment call for the researcher, the more important scenario 
issue is that such failures create a situation that must be 
handled with a great deal of tact during the rest of the simu- 
lator run, and during debriefing. Appropriate contingency 
plans should be made during scenario construction. 


Simulator Crashes 

Simulator crashes, including landing short during low- 
visibility approaches and overruns on short and slippery run- 
ways, can be in the same category’ as poor performance and 
need careful, reasoned consideration. Some researchers, 
believing the simulator should not be allowed to crash, will 
stop the simulator to prevent a crash, and then blame the 
inciderr on a simulator problem. Whether or not this proce- 
dure is desirable, it is not always possible, particularly in a 
low approach, aborted takeoff, or landing overrun situation. 
Fortunately, many incidents can be treated as minor, but 
controversy remains on the issue of whether or not to let the 
simulator crash. 

One of the characteristics of LOFT is the lack ot any 
intervention or interaction by the instructor or observer. 
LOFT flights are not interrupted for any reason, and con- 
tinue to their completion up to and including realistically 
simulated crashes, if that would be the operational outcome 
from similar performance in the real world. FAA require- 
ments for Phase II and Phase 111 training in simulators 
include “...the sound of a crash when the simulator is landed 
in excess of landing gear limitations. 

Lauber and Foushcc (1981) have cautioned that an acci- 
dent’ should never be the inevitable outcome of a (LOFT) 
scenario, although it is always possible that one will occur. 
They < Iso have noted the observation of airline training man- 
agers that “If an accident does occur during a LOFT session, 
it may provide the crew with a vivid learning experience.” In 
the military, simulators are used for combat training and 
getting shot down or crashing is not an uncommon experi- 
ence; however, military combat pilots know the risks and are 
prepared for them. The civilian pilot population does not 
have the same attitudes, values, or mission. 

Except on rare occasions, a crash should never be the 
planned outcome of a research scenario. Unfortunately, an 
unwanted crash can occur in LOS research, as it can in 
LOFT. If it docs happen, the simulated crash can create an 
additional problem for the researcher who is interested in 
creating as close to a real-world environment and reaction as 
is possible. 

In the real world, “postaccidcnt anxiety syndromes” have 
resulted from the acute situational anxiety which sometimes 
arises when a flight crew member survives an accident, and 
particularly one in which there were fatalities. The results 
can be severe. There are cases (sec Popplow. 1984. “After 
the Fire-Ball”) in which postcrash anxiety became so dis- 
abling that pilot careers were forced to be terminated despite 
psychiatric counseling and acceptable postcrash demonstra- 
tions of pilot proficiency. 

Wv found only one reference to a potential psychological 
or psychiatric problem associated with simulator crashes 
(Lager. 1965). but the increase in simulator realism, strong 
ego involvement of professional pilots in their performance, 
and ihe LOS practice of not interfering even if the simulator 


is about to crash, suggests that postcrash anxiety could arise 
in susceptible subjects. The possibility of a postcrash anxiety 
syndrome raises ethical and legal questions about the respon- 
sibilities of both researchers and research sponsors to protect 
the well being of human subjects. 

Legal opinion is beyond the scope of this report or the 
competence of the authors. The researcher should review 
Federal Regulations on the use of human subjects for 
research, and obtain legal counsel, in general, the regulations 
require that all human subjects must be volunteers; that the 
risks must be defined and made known to them in advance; 
that volunteer subjects may withdraw from the experiment 
at any time; that adequate safeguards and facilities must be 
provided to protect the subjects; and that the research must 
be conducted so as to avoid all unnecessary physical or 
mental discomfort, suffering or injury. 

The important point is that each researcher should be 
aware of this issue, and must decide whether or not a poten- 
tial crash is to be allowed to continue to its conclusion. Each 
case will have to be decided on its merits with something 
very close to an instantaneous decision. If a crash is to be 
permitted, the subjects should be prepared for it during their 
indoctrination, and the organization performing the research 
should be prepared to handle a postcrash anxiety syndrome 
if the crash cannot be prevented. If crashes are to be 
diverted, then all possible conditions which might lead to a 
crash will have to be known, procedures and scripting devel- 
oped to handle the problems, and the scenario team will have 
to be trained to recognize the situation quickly and execute 
the recovery procedures. 


Simulator Problems 

Simulator problems, including those induced by the 
research team or a simulator failure, are much more likely 
than a crash. One major reason for operator team training 
and simulator shakedown runs before the experiment is 
started is to minimize these kinds of occurrences. Simulator 
maintenance records should be reviewed to determine the 
most likely failures. The recovery procedures should be part 
of the scenario, which may have to be altered as unexpected 
problems surface during the shakedown runs. Providing for a 
greater number of trials than the absolute minimum required 
for the study is one way of coping with these issues after all 
methods of circumventing them have been exhausted during 
study preparation. 


DEBRIEFING 


Debriefing of the flight crew is an important part of LOS 
research. It is important to the crews who are understandably 
curious about their contribution and performance, and it is 


important for the researcher because this is the optimum 
time to get reasonably uncontaminated subjective data from 
the study participants. Debriefing is the best time to discover 
the covert thought processes behind the operational decisions 
made. 

Although debriefings can include structured or unstruc- 
tured interviews, postflight questionnaires, video tapes of 
crew interactions, and so forth, they should start with an 
open-ended review of the night by the flight crew itself. It is 
important to get their first impressions and overall reactions 
before specific research areas or audio or visual playbacks are 
discussed. Although it is impossible to avoid a certain 
amount of trying to “please the researcher” (who may be 
viewed by many of the participants as a prestigious authority 
figure), this tendency can be minimized if it is made clear to 
the flight crew that the researcher considers them the opera- 
tional experts and wants and needs their expert opinion. 

Researchers should remember always that the flight crew 
participants usually are, in fact, bona fide SMEs who have 
had an opportunity to view the simulator exercise from an 
important vantage point. The postflight debriefing is the 
ideal time to get crew reactions to the simulation and the 
scenario, and to explore the reasoning they used in reaching 
the decisions made during their flight. Audio and visual play- 
backs of the exercise are an effective method of providing 
“base points” and “reminders” for this part of the 
debriefing. 

There are advantages in having a full crew debriefing, so 
that crew interactions can be observed and consensus can be 
achieved. In many cases each crew member will have a 
slightly different view of the events. On the other hand, there 
are also advantages to individual debriefing — the principal 
advantage being that the results will not be dominated by 
the strongest personality. Both practices have been used 
effectively. 

A disadvantage with the isolated interviews is the addi- 
tional research personnel that may be required to interview 
two or three crew members without forcing some pilots to 
simply wait their turn until a preceding interview has been 
completed. It is not easy to do this without adversely affect- 
ing the quality of the succeeding interviews. In some cases, it 
may be possible to mitigate this problem by having the pilot 
waiting to bo interviewed till out a postexperiment question- 
naire on the scenario which has just been completed. 

Finally, and the point is worth reemphasizing, airline 
pilots are professionals who have understandable sensitivity 
regarding their reputation. Positive aspects of their perfor- 
mance should be reinforced. They should be thanked for 
their contribution, it is particularly important not to infer to 
any crew member that they have performed poorly, or that 
they had more problems than others have had. 

Poor or below average performance may need a rational- 
ized explanation. “Simulator problems,” or an allegedly 
“unrealistic scenario,” sometimes can be used to help explain 
this very sensitive area. If possible, subjects should be 


28 



promised that the results of the study will be mailed to them 
when they are published. They should leave the facility with 
a positive reaction to the research project and a feeling that 
they have made a positive and professional contribution to 
research in aviation. 


LEAD-TIME CONSIDERATIONS 


An unfortunate fact, well known to those who have done 
LOS research, is that the lead time required to prepare for 
the study almost always will be underestimated because of 
the number of potentially critical considerations that are 
beyond the researcher’s direct control. Preparation for the 
study can take much more time than it takes to run the 
experiment. The following list, and it is not a complete list, 
shows the kinds of items which should be considered. Eaeh 
of them can take a substantial amount ot time. 

1 Development of the scenario and the training and coor- 
dination of the scenario development team. (Sec appendix G 
for an example of the time that can be required, even with 

highly experienced personnel.) 

2. Interface with the employer and the pilot representing 
organization of the selected pilot population, if required. 

3. Production of the scenario script. 

4. Development and procurement of background environ- 
mental materials. 

5. Procurement of manuals, forms, and trip paperwork. 

6. Simulator scheduling. 

7. Procurement and installation of any hardware or simu- 
lator software changes required. 

8. Procurement and installation of data collecting devices. 

9. Development and testing of data collection and evalua- 
tion materials. 

10. Training of the experiment support team. 

11. Scenario testing and revision (includes data 
collection). 

12. Scheduling of pilots and outside support personnel. 

The actual amount of time that should be allocated to 

accomplish each of the foregoing tasks can vary tremen- 
dously depending upon the study requirements, the facilities 
available, the make-up of the research team, and the familiar- 
ity of its members with line-oriented, full-mission behavioral 
research. There is a very high probability that the prepara- 
tory steps will take considerably more time than was initially 
allocated. 


PREIMPLEMENTATION CHECKLIST FOR LOS 
RESEARCH 


This chapter has discussed many of the practical issues 
involved in conducting LOS research, particularly scenario 


and scripting requirements. The following is a checklist of 
items that should be considered in LOS research studies: 


Conceptual Stage 

1 Clariry the operational (or practical) problem. 

2. Define the research objectives. 

3. State the research question in researchablc terms. 

4. Determine the data needed. 

5. Select methods that will obtain the data needed. 

6. Determine the level of fidelity required. (Do you really 

need LOS?) 


Development Stage 

1. Determine scenario elements. 

2. Procure scenario development team. Supplemental 

expertise may be needed in: 

a. Operation to be simulated (type and location). 

b. Aircraft characteristics. 

c. Air traffic control. 

d. Simulator characteristics, 

c. Writing scenario scripts. 

3. Develop and test data collection and evaluative 

materials. , . . lt 

4. Select research operating team. Individuals/skills 

needed: 

a. Experiment coordinator (“wagon boss’ ). 

b. Pilot familiar with the aircraft and its operations. 

c. Operational expert familiar with operation simu- 
lated and its ancillary services. 

d. Data collectors. 

e. Observer (researcher) in cab -- may also have to 
operate simulator. 

f. Air traffic controller. 

g. Simulator operator. 

h. Simulator technician. 

5. Develop scenario: Select operational problems that can 
be expected to produce the desired behavior. 

6. Wiite scenario script. 

7. Administrative tasks: 

a. Determine simulator availability. 

b. Procure relevant software. 

c. Procure and install necessary hardware. 

d. Procure background environmental materials. 

e. Procure and install data collecting devices. 

f. Interface with employer and pilot representing 
organization if required. 

8. T ain research operating team. 

9. Determine subject pilot requirements (qualifications 

and numbers). . 

10. Determine training (including differences training) 
and indoctrination requirements for pilot population selected. 


rTrr-, 
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1 1 . Test and revise the scenario. 

1 2. Develop pre flight briefing material and protocol. 

13. Develop postflight debriefing material and protocol. 

14. Determine performance measurement requirements. 

15. Determine data reduction and analysis procedures. 

16. Develop data reduction and analysis software. 

1 7. Perform sensitivity analysis of measures. 

18. Schedule pilots. 


A FINAL COMMENT 


Past experience demonstrates that once a decision is made 
to conduct LOS research, few short-cuts can be taken. Line- 
oriented simulation research is both equipment and labor 


intensive, and takes several months of calendar time. A com- 
plex LOS study easily could take more than a year from 
inception to completion. More often than not. the require- 
ments of time and resources will be underestimated by all 
but the most seasoned LOS researchers. LOS is an exercise in 
details, all of which are important, and many of which can 
compromise the results of the study if they are not attended 
to with accuracy and precision. 

However, good LOS research is well worth the efforts 
required, it is providing new insights into critical areas that 
many believed were not researchable, including many behav- 
io rally related issues that have been called the “last frontier” 
in air transport safety. Conducting such research is a chal- 
lenging task and there are substantial rewards for doing it 
well. The challenge to the researcher is limited only by his or 
her imagination. 
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CHAPTER 4 


research recommendations 

full-mission research validity testing 

INTRODUCTION 


The purpose of this chapter is to recommend rescarcli to 
improve full-mission LOS research, and not to address the 
broad issues of simulation and training. A thorough review o 
simulation issues from the perspective of the behavioral 
sciences was conducted by the National Research Council, 
Committee on Human Factors, Working Group on Simula- 
tion (Nones et ah, 1985). A study of simulator training 
requirements and effectiveness was conducted by Semple 
ct al. (1981). These reports address major issues in simulation 
and, especially in aircrew training, should be consulted for 
broad issues and research recommendations. 

After examining the techniques of LOS research discusse 
in the previous chapters and in appendix C, it became evident 
that we need to know more about the use of LOS in behav- 
ioral research. Increasing the available knowledge can lead to 
increased confidence in conclusions drawn from LOS 
research, as well as to increased research productivity. The 
following key issues have not been answered well to date. 

1 Full-mission research validity. Does LOS that is con- 
ducted in accordance with the guidelines presented in this 
report produce an environment in which the exhibited behav- 
ior is the same as would be exhibited in the real world? 

2. Alternative forms of valid simulation. Can the behavior 
currently studied with LOS research be produced with 
abstract or part-task simulations? Stated another way: What 
are the research criteria for using LOS, and are there ways o 
obtaining valid behavioral measurements at less expense, 
with greater efficiency and control: 

3. Optimization of full-mission research. Can LOS 
research methods be optimized? Are there alternate methods 
to achieve greater realism? 

4. Human performance development. Can current ana* 
lytic tools be improved, or new ones be created for assessing 
LOS features for a given application through the develop- 
ment of models of human behavior? 

5. Subjective measures of fidelity. Can the elusive quali- 
ties of fidelity and validity in LOS research be measured by a 
simple, direct, and efficient method of subjective measures 
derived from expert personnel who serve as LOS research 
subjects? 

6. Integration of research efforts. Can several of the 
above areas of research be studied together? 


Problem 

In spite of extensive efforts to achieve high fidelity, the 
possibility remains that the behavior exhibited in the simula- 
tor is not the same as the behavior that would have occurred 
in actual flight. In fact, line pilots serving as subjects occa- 
sionally indicate that they would have behaved differently in 
the airplane. Although there is a suspicion that this some- 
times may be a rationalization for poor performance, it 
might NOT be. Expert pilot comments may be indicating a 
simple truth and a simulation deficiency, whether or not that 
deficiency can be articulated clearly. Whatever the case, 
researchers should know the limits of their tools. 

Even with optimum fidelity, a subject flight crew can 
never be expected to think they are about to fly a line flight 
while they are climbing the stairs to enter a research simu- 
lator. It is not clear at what moment (il ever) the crew mem- 
bers become so engrossed in the simulated problem that they 
have totally forgotten that they are in a simulator. Complete 
preoccupation with a challenging simulator task does not 
guarantee that all aspects of the preliminary steps, including 
the drive to the simulator location instead of to an airport, 
will stav obliterated in the pilot’s memory. The behavioral 
implications, if any, of these questions are not known. 


Approach 

One ambitious approach to answer these questions is quite 
clear Fly identical full-mission scenarios in both the aircratt 
and the simulator, collect a battery of measurements, and 
test the degree of correlation between the measures taken in 
the two environments. A major problem, aside from the 
enormous expense involved, is that of collecting in-flight and 
simulator measurements under identical scenarios. 

Since it is not possible to specify in advance all the details 
of an actual flight to correspond with a scenario designed for 
the simulator, one must first measure, and record during 
flight, all basic parameters including communications, 
weather, and so forth, and then attempt to fly an identical 
scenario in the simulator. This process can be continued until 
a representative and sufficient sample of flights are recorded, 
providing a set of scenarios for duplication in the simulator. 
This would provide a paradigm similar to that of a backward 
transfer of training study (table 4-1 ). 
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There are a number of practical and methodological prob- 
lems associated with such an undertaking. One is the diffi- 
culty of recording data during actual flights, and of achieving 
combinations of malfunctions, weather, traffic, and so forth 
that produce a wide range of behaviors. The task of data 
recording for a broad range of behavior could be impossible 
without using a special instrumented aircraft (or an elaborate 
video tape and manual data reduction effort). A large num- 
ber of flights might be required to achieve data collection for 
a desired range of behaviors. 


TABLE 4-1 COMPARISONS TO VALIDATE FMS 
RESULTS 

Simulator 

Crew 1 * Crew 2 Crew N 

XXX 
X x X 

XXX 

*Same Crew 

A methodological problem is created if the same crew is 
used for both flight and simulator because the simulator per- 
formance will be contaminated by the previous flight experi- 
ence. For example, would the same decision making have 
occurred without knowing what had happened during the 
prior actual flight, or would different decision making occur 
because of “lessons learned” about what had happened 
previously? 

Once the simulator scenario is established, data can be 
collected with a number of simulator crews. However, since 
people are unique, one would again expect a wide range of 
behavioral outcomes because of intersubject variability. 


Aircraft 

Scenario 

Crew 1 * 

1 X 

2 X 

3 X 


Expected Results 

Data would be produced which should reflect on the 
validity of FMS research. Serendipitous information on how 
to improve FMS research also may be secured. The magni- 
tude of the required effort is unquestionably large. This 
recommendation is made only because of the potential 
importance of the results; we may never know the validity 
of simulation-study results until such an effort is undertaken. 


Alternative Approaches 

Because the recommended effort is so potentially expen- 
sive and might never be done for that reason, research should 
be conducted to find ways to approximate such a study. As 


an example, there arc many aviation incidents every year, as 
well as accidents; these events are documented in data bases 
such as those maintained by the NASA Aviation Safety 
Reporting System (ASRS) and the National Transportation 
Safety Board (NTSB). These sources could be screened for 
types of events that have occurred several times, and those 
types of events could be placed in scenarios in simulators, if 
the response to the realworld events were replicated in a 
simulator with new crews (using similar equipment, operating 
procedures, crew experience, duty time, schedule constraints, 
and ATC and meteorological environments), there would be 
some evidence that performance in the simulator was repre- 
sentative of real-world performance for that type of event. 

This alternative approach would require a substantial 
amount of work. The circumstances surrounding highly pub- 
licized incidents and accidents might be recognized by the 
crews, and their behavior might be altered by their prior 
knowledge. The unpublicized incidents (or published back- 
ground data) might provide the most fertile source. One 
would need to enrich the database information with details 
which would be sufficient to construct a scenario event; a 
call-back to the crews involved (or a follow-up questionnaire) 
would be needed to derive such data. Also, incidents would 
be a better source of data because the flight crew members 
are still around to help recreate the scenario. Then a scenario 
might be constructed to include the circumstances which led 
up to the event in much the same way as LOFT scenarios, 
which are often derived from events that really happen in 
airline operations. 

There are methodological issues that would have to be 
addressed, such as (but not limited to) how' many occur- 
rences of similar (to real-world) behavior would be needed to 
conclude that the behavior didn’t occur by chance alone? 
Conversely, what conclusions would be reached if none of 
the simulator crews replicated the real-world performance? 

It is not our purpose to fully develop such a method. We 
suggest that there might be some practical ways, short of the 
scientifically best way we have recommended, to demon- 
strate that performance in the simulator is representative of 
real-world performance for specific classes of events. 


ALTERNATIVE FORMS OF VALID SIMULATION 


Problem 

There is an informed (but unproven) consensus that 
high-fidelity FMS techniques provide insurance against the 
possibility of conducting research on behavior that is sub- 
stantially different from that which would occur in the real 
world. It is believed that this approach ensures that the 
human operator collects and processes information in the 
same way as in the real world. Its substantia] advantage is 
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that it avoids the difficulty and almost prohibitively expen- 
sive task of performing complex behavioral research in the 
real world. 

Unfortunately, research productivity can be quite low 
since high-fidelity, full-mission research is time-consuming 
and costly. It is also always possible that the full-mission 
approach will include a host of extraneous factors that may 
mask the results oi interest. Finally, at some time the interac- 
tion of effects must be studied, not just results embedded in 
a typical situation. In this case, there may be a need for 
simple and truncated environments for comparison with full- 
mission results. In short, there may be types of studies which 
can be accomplished better and more efficiently using other 
research methods, including less than FMS. The problem is 
that researchers today are on uncertain ground in many cases 
and, understandably, elect the safer route — maximum fidel- 
ity in a full-mission context. 


Approach 

When using less than FMS, it is NOT necessary (although 
it might be desirable) to compare results to real-world 
in-flight performance, since performance in the high-fidelity, 
full-mission simulator can be used as a baseline or criterion. 
The a priori assumption is that the high-fidelity, full-mission 
simulator does produce the equivalent of real-world bchuvioi . 
Therefore, the approach is to perform a high-fidelity, full- 
mission studv, and then use a number of levels of simulation 
with the same purpose or subpurposes. 

The final step would be to correlate the results to see if 
equally valid data was obtained using simpler devices. The 
degree of comparability, of course, will depend on the nature 
of the behavior involved. While control behavior may not be 
measurable with all abstract simulations, it is conceivable 
that all forms of decision behavior may be measurable using 
significantly lower levels of abstraction. 

This approach is a variant of the approach used success- 
fully by the Visual Technology Research Simulator (VTRS) 
behavioral research program at the Naval Training Equipment 
Center (cf. Lintcrn, Wightman, and Westra, 1984). In the 
VTRS program, transfer effects of reduced visual system, 
simulator, and motion system fidelity arc estimated by a 
“quasi-transfer” study which measures the transfer of train- 
ing within the simulator from the reduced simulation to the 
highest-fidelity simulation. The approach permits many fac- 
tors to be screened in the simulator environment, and both 
transfer of training and performance effects can be measured. 

It is recommended that a matrix of alternative simula- 
tions be used, as shown in table 4-2, with levels of abstrac- 
tion as one dimension of the matrix, and part/full-mission 
simulation as the other dimension. A large number of levels 
of abstraction are possible, including high-fidelity simulation, 
simulation without visual and motion, general aviation simu- 
lator (e.g.. GAT-1 class), microcomputer with CRT display, 


keyboard, and joysticks (with approximate aircraft dynam- 
ics). microcomputer with CRT pages and keyboard (no 
dynamics), and pencil and paper. It is believed that full- 
mission and part-mission simulations are possible with each 
of these devices. The part-mission simulation would treat 
only key portions of the mission, within or out of the con- 
text of < n entire mission. 

The highly abstracted simulations require some amplifying 
description. The CRT/JOYSTICK simulation assumes a 
rough model of aircraft dynamics; a display of instruments; 
joysticks to control pitch, roll, and thrust; and a keyboard to 
control avionics, aircraft configuration (gear, flaps, and 
spoilers i and subsystems (electrical, hydraulic, pressurization, 
and so forth). An improvement of the popular “flight simu- 
lator” software for microcomputers is envisioned. 


TABLE 4-2. COMPARISONS OF ALTERNATIVE 


LEVELS OF 

Isolated 

segments 

SIMULATION 

Connected 

segments 

Full 

mission 

Full-mission 

X 

X 

X 

Fixed no vis 

X 

X 

X 

General aviation 

X 

X 

X 

CRT, joysticks 

X 

X 

X 

CRT keyboard 

X 

X 

X 

Pencil/paper 

X 

X 

X 


Each member of the crew could have a microcomputer 
system or terminal, and a flight or flight segment could be 
flown by a single pilot or a crew using the devices and what- 
ever ATC communications and ancillary services are needed. 
State-of-the-art microcomputers, networked where necessary, 
could provide all the capability needed for airports, radio 
navigation aids, scenario control, and performance measure- 
ment at a fraction of the cost of an FMS. 

The envisioned CRT/KEYBOARD simulation would pre- 
sent a page of text and/or graphic information. The user 
would be present with various options, including decisions to 
take action or requests for more or specific information. 
Based on the user's action, another page of information and 
options would be presented. The experience would be analo- 
gous to reading the script for a play, but with the possibility 
of the play branching out in many directions depending on 
the actions of the reader. 

Either booklet or interactive computer media could be 
used, with the user being directed to turn to a designated 
page with the booklet, or a new display being generated auto- 
matically with the use of a coniputci. The computer imple- 
mentation would have the advantage of allowing complex 
algorithms for determining the next display, and also would 
allow automated measurement of user selections. 
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Full crew participation with multi-terminal configurations 
are possible with a computer implementation. The users 
could be "walked through” an entire flight, step by step, in 
the context of an FMS. For part -mission simulation, the 
users could have a similar treatment, but for only a portion 
of the mission beginning with a display of the initial condi- 
tions for that segment. 

Expected Results 

It is expected that this avenue of research would show 
that some forms of study can be conducted more economi- 
cally with little disadvantage compared to high-fidelity, full- 
mission research. This could result in greater research produc- 
tivity. On the other hand, if the results indicate the necessity 
for full-mission research, that level of research productivity 
must be accepted, and any extrapolation of laboratory 
research in those areas must be suspected. 


OPTIMIZATION OF FULL-MISSION RESEARCH 


Problem 

Two additional problems can be addressed as corollary 
activities in any study of alternative forms of simulation. 
The first ot these three problems, which are actually a set of 
problems, occurs because of the multitude of decisions 
required in the design of LOS, full-mission research. For 
example: 

1. Is a day-night visual system required, or will a night- 
only visual system suffice? 

2. Should there be multiple air traffic controllers (and 
different voices), or will one professional air traffic control- 
ler suffice? 

3. Is anything really gained by beginning the mission with 
a long low-activity (i.e., normal) segment, or can one go 
directly into high-activity segments which will provide the 
primary research data? If so, should an initial portion of the 
high-activity segment be considered a warm-up? 

4. Will broad classes of malfunctions provide essentially 
the same decision and crew coordination tasks? 

5. Are there scenario-independent measures of behavior 
which would be valid and meaningful for research? 

These kinds of questions raise the possibility of general- 
ized scenarios and measures. The questions arise because, in 
cockpit resource management training, the specific system 
failures and scenario events arc not as important as the fact 
that the scenario produces a complex problem for crews to 
solve. The flight crews have to exercise good resource man- 
agement skills to solve the problem safely and efficiently. 
Traditional pilot-system performance measures of deviations 


from a known profile capture only a small part of the impor- 
tant behavior. 

The second problem occurs because of the extensive num- 
ber of precxperimental missions which must be flown before 
a complete and refined scenario is developed. This is time 
consuming and expensive. The preexperimental testing may 
be as much as half the total effort. Attempting to shortcut 
preexperimental testing, which can be a very great tempta- 
tion, is likely to result in many surprises during experimental 
testing. The result may be that much of the data collected 
has little or no meaning. 

Approach 

The first problem can be treated by experimentally com- 
paring alternative forms of high-fidelity FMSs. For example, 
a scenario can be done by starting with a low-activity seg- 
ment, and then repeated by beginning with the high-activity 
segment that is expected to provide the primary research 
data. 

The second problem may be approached by using low- 
fidelity abstract forms of simulation (perhaps the booklet or 
computer form) to implement the detailed scenario for 
review by expert flight crews. For example, an entire sce- 
nario can be documented in booklet form, and then com- 
ments can be collected from SMEs. Revisions can be made to 
the booklet, and then more SME data can be collected. It is 
possible that refined scenarios could be quickly and inexpen- 
sively derived in this way. 

The booklet approach can provide leads to the measure- 
ment issues, but it is unlikely to solve them. Behavioral 
measurement issues require continued effort and research in 
areas which are under investigation by NASA at this time 
(e.g., workload, communications, and performance measure- 
ment), and in human-performance model development. 

Expected Results 

More refined and efficient forms of FMS should result 
from this study. The study could lend credibility to FMS 
results and give insight into key scenario-design components. 

HUMAN PERFORMANCE MODEL DEVELOPMENT 

Problem 

Full-mission research attacks more complex aspects of 
human performance than have been treated extensively in 
the past. Included are the domains of human performance 
termed supervisory control and cognitive processes. It is 
highly probably that a model reflecting a better 
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understanding of human behavior, even though approxima 
and incomplete, can be used to improve the design of FMSs. 

For example, at the level of simple task analysis, a knowl- 
edge of the stimulus and response requirements for a given 
task can be used to determine if the simulation would per- 
ni ,t behavior as it occurs in the real world. If the simulation 
does not provide the information or permit the response ic 
model indicates is needed, then one may conclude that real- 
world behavior is not likely to occur. This is an example of 
the classical mixing together ot empirical and theoretica 
attacks to form a synergistic relationship. 


obtained subjectively from expert users. The expert us 
such as the experienced line pilot, is the possessor ot detailed 
system-specific knowledge and experience - a level of knowl- 
edge and skill not easily acquired by a task analyst. 

"Ostensibly, the expert user can compare the FMS will 
prior experience and evaluate the simulator capabilities. A 
judgment can be elicited as to the estimated differences 
between simulator and night behavior. An instrument is 
required which will do this in a valid and reliable way. No 
other approach is as tractable as subjective measurement with 

the expo t user. 


Approach 

Two areas of model development which appear to promise 
important insights into the human performance involved in 
FMS are Supervisory Control (Sheridan, 1983) an tie area 
of artificial intelligence called Expert Systems (Crowe et al 
1981 : Obermayer et al.. 1984). Both are especially important 
to the design of FMS, even when the research is directed 
toward other specific issues, such as the social interactions 
of crew members. 

The supervisory control approach can structure and iden- 
tify' the multi-level control tasks of the crew. Analysis o 
cognitive processes in terms of models used for expert sys- 
tems will allow extraction of specific rules used by crew 
members in making inferences and taking action. Expert 
systems have been developed for an array of sophisticated 
applications. They have considerable promise for a muci 
needed representation of decision behavior. 

Extraction of data for model development during FMSs. 
and then a subsequent testing of the models against human 
performance in other FMSs, is the recommended approach. 


Approach 

A structured questionnaire must be developed which can 
ask detailed questions about each of the features involved in 
the simulation across each segment of the mission. The fidel- 
ity of the simulated task can be compared with a subjective 
judgment of the level of fidelity required to perform the 
task, as it is done in the real world. The development and 
testing of such an instrument is substantial but can t 

accomplished. , .. . * 

For example, each question must be carefully phrased 

and tested for correct interpretation by the appropriate pop- 
ulation The effect of the order of questions and orientation 
of scales can be moderated through randomization (suggest- 
ing computer implementation). Data reduction and analysis 
software can be developed using algorithms for attitude mea- 
surement to improve interpretation of results. Thorough 
validity testing should be accomplished by comparing the 
subjective measurement with other corroborating data. 


Expected Results 

The short-term result expected is a specific framework 
for describing human behavior in FMS. This framework 
would focus attention on the behavior which the crew 
exhibits the manner in which it is exhibited, and the result- 
ing correspondence to real-world behavior. Improved under- 
standing of this behavior will have many benefits, including 
improved analytical tools for the design of full-mission 
research studies. 


SUBJECTIVE MEASURES OF FIDELITY 


Problem 

There is a need for a quick and easy test for satisfactory 
levels of simulation fidelity. Such information may be 


Expected Results 

Given appropriate development, a reliable and valid 
instrument is expected to result. It is an instrument which 
can be used easily, and can be expected to achieve wide- 
spread use. Therefore, it is important that an appropriately 
designed and tested instrument be developed before an 
incompletely developed method becomes a defacto standard. 

INTEGRATION OF RESEARCH EFFORTS 

Although specific areas of research are discussed sepa- 
rately in the preceding sections, they need not be s u ic 
separately since studies can be designed to address two or 
more of these topics at the same time. For example, the va fi- 
liation and alternative levels of simulation problems can be 
memed' the LOS research data collection could be used tor 
comparison with in-flight data as well as with data from 
abstiact simulations. 
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The data collection for comparison between alternative 
levels ot simulation can be combined with data collection for 
model development (with data collection for both purposes 
beint- derived from the same subjects). The subjective mea- 
surement development can take place with any of the other 
simulation studies. Furthermore, an experimental design can 
be developed which combines the study of alternative levels 
ot simulation with alternative factors for optimizing LOS 
research. 


CONCLUSION 


A detailed plan of research depends upon the tradeoffs 
between available resources and competing goals. While such 
an attempt is outside the scope of this study, we believe that 
the research recommendations presented here can provide 
presently unavailable and needed knowledge regarding the 
use of FMS in applied behavioral research. We also believe 
that these recommendations can improve the utility of LOS 
as a research tool. 
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FIELD INTERVIEW 

NA sa study of preflight and postflight operational 


PERFORMANCE 


Dr. H. Clayton Foushee 


research goal 


The purpose of this study was to examine the effects of 
duty-cycle exposure on an airline crew as a function o an 
actual line trip. Two groups of subjects were used: those who 
went directly to the simulator at the end of a 3-day trip 
(postduty ) and those who performed on the simulator after 
3 davs at home (preduty). An observer noted critical events 
and rated performance during each simulated night. Video 
tape recordings were made so that similar ratings coul c 
performed by a panel of experts at a later time. Critical flight 
parameters were recorded onto floppy disks from the simu- 
lator computer and were time-synchronized with the video 
tape recordings. 


naps cannot be raised fast enough to execute a missed 
approach. Additionally, there is reduced effectiveness of 
thrust reversing and the anti-skid systems. As a result of the 
malfunction, the crew must modify their normal approach to 
City C ; nd may declare an emergency condition. The final 
destination was characterized by hazardous terrain and a 
short wet runway which had additional implications lor the 
malfunction. 

Kev decisions designed into this scenario are. 1) req 
ing more fuel during flight planning: 2) requesting a take-off 
alternate, since immediate landing at the takeoft location is 
not possible; 3) determining that landing at City B is not 
suitable for Category 11 operation, because of the crosswind 
component; 4) determining that City C is the only suita e 
alternate; and 5) coping with the System A malfunction. 


GENERAL DESCRIPTION OF SCENARIO 


SPECIFIC SCENARIO CONSIDERATIONS 


It was desired to collect data in the context of a realistic 
flight rather than what may be judged as a contrived 
sequence of emergencies. It was desired to develop a scenario 
so there would be no question that the results could have 
happened during airline operations. Key decision points were 
designed into the scenario. Other conditions were defined so 
that generally the same flightpath should be selected by all 

Consistent with the availability of subjects and a simula- 
tor, a short-haul flight was planned to start at City A and end 
at City B The weather was generally bad. and the aircraft 
was heavy with minimum legal fuel. Some equipment was 
inoperative at the start. At takeoff time the airport would go 
below landing minimums. At City B the crew would find that 
conditions were not suitable for a Category II landing and. if 
a landing was attempted, they would find that the actual 
ceilings were below decision height. The only alternate desti- 
nation with acceptable weather conditions, given the fuel 

state, would be City C. , , 

A “System A” hydraulic failure would be introduced 
while executing the missed approach procedure at City B. 
This failure requires manual actuation of the landing gear an 
electrical actuation of flaps, which is very, very slow Also, 
the leading-edge flaps and gear cannot be raised once they arc 
lowered, and a 15°-flaps approach is required because the 


Subjects 

Arrangements had to be made with both labor and man- 
agement organizations to acquire subjects. This, together 
with the availability of a suitable simulator, narrowed subject 
selection to one airline. The scenario was therefore tailored 
to the requirements of that airline. Each subject was 
informed that involvement in the study would be anonymous 
and that data would be identified only by a code number. 
Otherwise, subjects could fear that they were being given a 
check ride which might influence their employment. 

All of the subjects were volunteers. Because ot flight crew 
scheduling realities and the nomadic behavior of pilots, it was 
often difficult to schedule full line crews for the simulator 
runs, despite a high level of interest and very good coopera- 
tion from the pilots. It was considered essential to have a full 
line c ew for each simulator exercise. Scheduling difficulties 
have prolonged the time required to complete this phase ol 

the stjdy. , . , . , 

Each crew was given a preflight briefing which stressed 

the importance of their participation. They were told that 
the study really depended on them. They were asked to role 
nlay - to fly the simulator and make any decisions exact y 
as they would on a line trip. Full dispatch and all other pre- 
flight ' services would be available, and they would be 
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expected to complete all preflight papers in the same fashion 
they are tilled out for line (lights. Care was taken not to tell 
crews how long the exercise would take since this would 
indicate that a normal (light would not occur. 


Observer/Ancillary' Support Personnel 

Individuals trained in weather, and ATC were used to 
make communications to the airline crew. A script was used 
to provide routine communications consistent with the 
scenaiio. as well as to simulate communications to other air- 
crews. Background communications tapes were piped into 
V HP 1 and 2 to simulate other traffic and ATIS information. 
Information was included in communications to other air- 
crews which might also be used in decision-making by the 
experimental crew. In addition to script communications the 
support personnel had to provide any information which the 
crew might request. An observer was positioned in the simu- 
ator flight cabin, and while this might present a deviation 
rom realism, this intrusion was considered necessary for 
data collection. The observer also provided functions for any 

other support personnel, such as passenger-cabin crew 
personnel. 


Flight Planning 


Simulator Equipment 

Some equipment was not available in the simulator, 
namely, radar, ACARS and the number 3 VHF. The experi- 
mental crews were briefed that this equipment was “inopera- 
tive. Modifications to the procedures were necessary as a 
result. The crew had to be briefed on alternatives to the 
number 3 VHF, since it was normally used for the acquisi- 
tion of ATIS. The crew had to advise ATC that radar was 
inoperative. Simulator malfunctions had to be identified and 
fixed during the ground flight-planning period. Experienced 
flight crews were used during pretest to ensure that simulator 
procedures and equipment were company specific and con- 
sistent with those used by the experimental crews during 
normal line flights at the airline in question. 

Everything about the flight was made as realistic as possi- 
ble. It started with a pushback after the cargo and main 
cabin door lights indicated dosed and the flight crew 
received an appropriate message from the cabin crew and 
clearance from the ground. The timing of the pushback with 
activation of simulator motion jars the simulator to realisti- 
cally suggest a pushback with a tug. 

Taxiing in the simulator presented a problem since the 
visual system did not have sufficient field-of-view to permit 
right turns. The scenario had to be designed so that only left 
turns were required for taxiing. 


Actual company trip paperwork and preflight plannin 
were performed as they would be for a normal flight. A 
experienced dispatcher was available. The first decision fo 
the flight crew involved the flight plan. They had close b 
minimum weather, a heavy airplane with a lot of pavload 
and planned fuel that was legal but less than they wen 
normally used to taking. They could add fuel but the dis 
pa teller discouraged this because it would reduce their pay 
load. There was a developing front along their route which 
was causing rain and generally low ceilings throughout the 
area. In addition, the copilot was able to start role playim- bv 
setting up a “cold” aircraft. ' 


Weather 


Care was taken to ensure that weather could have been 
typical of weather previously experienced in the scenario 
area for spring through fall. Experienced weathermen were 
consulted. It was desirable to channel the flight to City C so 
that experimental flights were controlled to a common 
lghtpath; also, it was judged by weathermen to be typical 
of the area that City C could be “open” when the rest of the 
area was below landing minimunis. Lightning flashes, turbu- 
lence and rain showers were used in the visual scene to alert 
the flight crew and corroborate with reports of deteriorating 
weather conditions. 


High-Low-Workload Periods 

The scenario design produced a flight which started with 
a low-workload flight and ended with a high-workload flight 
segment. This permitted an analysis of behavior during both 
high- and low-workload conditions. However, the initial low- 
workload conditions, typical of most airline flights, provided 
a period for the crew to develop a realistic mind set and alle- 
viate “simulator syndrome.” It was the belief of the investi- 
gator, based on past observations, that crews are generally 
suspicious upon entering the simulator and abnormalities 
introduced early in a scenario tend to reinforce these suspi- 
cions. By letting the crew relax, the probability of their 
behavior being realistic when a problem does occur is 
increased. Such judgments about realism as well as judgments 
about the level of workload, were made as the result of 
extensive pretesting flights. 


Malfunctions 

It was desired to include a malfunction which required 
ugh-levei decision-making but did not pose a serious hazard 
o flight safety. It was desired to end the flight in a safe and 
satisfactory manner. The chosen malfunction required time- 
consuming manual deployment of gear and electrical exten- 
sion of flaps. The malfunction was introduced while 
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Debriefing 

The crews were instructed not 
simulator experience with any other flight 
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SPECIFIC SCENARIO CONSIDERATIONS 


Simulator 
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Subjects 
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^r^;ta:^eas's :r o,. = 

over several years inviilving consirec^ concerned 

cuuipmcnt and rhe mrnurg of a icnti(ic mve s,i B a- 

“S,To l‘.&' “ f hU '” m 

irg with machines. renuired to solve every 

^is'lu^ror insumeienr. 

" The MC " n<1 




for obtaining it These steps are h • U)nS ’ der tlle alternatives 
are stressed here because if tl bV1 ° U , S a ' ld importanl - They 
done carefully, the end result preIlnilnar >' ste PS are not 
essary research! " bC ,nappropriate or unnec- 

t JI c , 

for a great deal of research F ' '' prob,em ls responsible 

-» *» Iree^perfbrmed L ZTSLS 

motion on perform i . er ect of platform 

w aag, 1981 for a review). YeT^Ms Simu,at « r (^c 
marshall strong support for eith, mtposstble to 

question of whether platform .notionTnw^fo^ 

Jar purpose. euea for a particu- 

rnJ™e d -yesC™;Ser Ml ’ C ' T* * "*« «• «w*r 

studies were different Lond "' ons in previous 

therefore the res “d. T. T '“T' 8 ° f ■"h 

The answer nri^Te 1 f ^ ®f pres «"< 
the difficulty fn dOTere ' „“t S 'd andfindS ' ,M 

can, downstream effects on reTea’S ELT Sig " ifi ‘ 

researr::,T £ ‘’‘"“'T P “' P - ° P "« 

applied behavioral research- i.-T !, malP pll rposes of 
exploration. and e ,, a lo „ F . i n KS,, " S ' probl ™ 
ingsnbsee,io„s. d '' 8 d,!CUSS « 1 ™ 'he follow- 


relationships in time of the associated" deSCnbe tlle 
of events. ec ^ actlons sequence 

f nr Kp, -“> * 

researcher shouid have ^o Li" S "“' d be - b “* p « 'he 
the causal or correlational i • ' preco,Ke * vc d ideas about 
This sometimes is difficult 00 ^ am ° ng the Variables - 
there will be at leLt a „ 0ti o ^ Cases 

conditions that are thouaht r°h road c,asscs of variables or 
data are likely to be "o , ° lmp0rtant ‘ Morc detailed 

Plhers. The researcher should be Z,‘!,7 Lfc'Y"" “r 
missing or underestim-.tinn ,r - 1 1 danger of 

.Noa t ,e„, P ,:rr™n~r reriic,urs ' 

situation in problem exploration The , h ° pCratlonal 
to discover the particular factors that are”" 1 ^ ° bjeCt,Ve is 
when the even, interest docs or dL 2 27 " ^ 

Evaluation 


Hypothesis Testing 

or • ,o cons “ 

'he relationships £2727£7n "l"'"” ^ 
research is to confirm nr m , T ° bject of the 

form of the nredirf s P rov e these predictions. The 

cause and effect The S ^ correlational or stated as 

are » me for ra „,,,,„r:f c tt;“r i T“ , ' s ' 1 " 8 

level of description in terms r yp 1 esis at some general 

hTo,erU^rnr™^ 


Research for evaluation is also atheoretir Itc • 

Evaluative 772272 77'° * 

comparisons of systems (^dementi J Ut . USUa,Iy inVoives 

attempt to discover performance differences^' ^ 
studies often are done tn find . r r cnces. Comparative 

merit, configuratton o T ^ ° ne type of equip- 
mance than ” Pr ° CedUrC rCSUltS ta bc “- Per- 
formance againTapI’edttLrned"^ ° f per ' 

dt: example"^ 

the performance of all normal tid ^ support 

fight tasks. tnergency visually guided 

is ,p pbiaip b -f« p o por- 

have adequate baseline data orV " 8 S>StCm WlUCh does not 
p «w or different 


Problem Exploration 

factors^ huii Tiuribute to "“"hi di “° Ver and isoli »' 

CX h bZ:sz “ T L:!: ' ip ~ ^ 

principal objeeii.es of ,,ob,e m 


application context 


7 s : r r:r;i - ^2772 

display mechanisms. Behavioral research ^ C ° ntro1 3nd 
mined by the role of the i search topics are deter- 

systems and by the practicalT^I ° T modified 
> practical problems that arise during their 
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development end employment Mr .he "« ^ 
briefly review the nature ° f °'* ' “"j,, ,„d the 


Older Systems 


An older system. 

lions, is an “1" Vnd controls. Each control and 

with its own set of display relation- 

display is dedicated to ■ _ has a simple, 

ship to its subsystem. A Similarly, a display 

Si:r;r.re;r:«y C da.a ahon, some aspect of 

• he , Sterns, the hnm : o r ».- = ^ 
closing the control loop et subsystems to 

as well as the coordinated ope. <•■»»,* ” * f peoplc 

achieve the general Sy *'“' J.,em is directly ptopottional 

rri:x::f^£c : re«^ 

F „r example. /^°! d l" r ge 0n des„latc geographical areas free 

Z membe" Z retired «**«• «* 

navigator and radio s ,ents (lnc l„diitg aircraft) 

Human operators ot olde y information 

fniftl, their toles b, or very 

from several discrete son-c s and making ^ ^ henM 

frequent control aetna io ■ contI „l, procedural in 

their training, are basic 3 the ope rators to 

orientation. Manual eon , t o control the system 

learn the system dynamics we ® d Aircra ft configura- 

with the precision and stabd > £ dcctncal hydraulic, 

tion and subsystem op of tasks that are essen- 

and a v i < j ti ic s sy s te m s ^ a r e an^x am ^ ^ and abnorma l 

nTode^is largely a matter of following a set of rules o t e 

form, “if x then do y. distributed 

Olde, complex ' f “^„dy ” qnire a manage, 

among several operators. They q V ^ such as the 
who also may have an aedree t^ in fotmation from the 
captain of an air , es their activities, makes 

other personnel directs a P iblc f or the 

* ”** ; e : iS ' 0nS ' XLS : on ,h« particular hind of 

:;^ir^r^r:; t ,u to conn„ed,o 

specific tasks. roinoved from moni- 

toring and control of h j ^ control bctwce n the lowest 

Sr"nd “lent tnanaget also will be filled by humans. 


This has advantages and disadvantages lb, respond, ng 

of-tolcranceot emergency co, td.ttons- e ,,, 

A principal advantage rs “ 1 J“ ,„ n lhelr span of 

delect and respond to a nor information to the 

authority ot at least communicate status rnM ^ ^ 

manage' ^“‘^‘'"'" tremate d.strtbutcd at a single level. 

^r,n.etmcdiare, r l ; ofe = . r ^d = 

tothemanagman be sueelne, and desettp.t.e 

of the exact fault. systems are that 

Son, serious disa«s^t ese^o^n imp endin g 
an operator .nay not n J or 5e able to take a 

failure, be able to react quick y J $ of authority . An 

required action out o is located but separated in 

event affecting several system , c f control. In 

control, may require actron at a wiU haVC t0 
such cases the intermediate or * the comm0 n basis 

integrate the individual reports, d e 

of the problem, decide on. ^action was 

and follow up to Ik ^ cer am in a timely and effec- 

taken. It is not always easy w 
tive fashion. 


Newer Systems 

me trend of new developments ita 
twofold, 1) functional mtegrati J J C vels with a 
2) implementation of J“^" d rollcr overseeing the full 
high-order, co.nputer-b $ ^ of the human is to excr- 

range of system func ■ m Emphasi/ .ing the role of 

else overall contro o ^ ca Ued supervisory con- 

the human, systems of this >P The general 

uo. systems lnnc.ton.1 

ut thv 1-1 (Sheridan e, ah, 

'“nle physical atchitcc.nte ™ V j"; u °‘ V c ; ^ 

and diffet ."“"S 7'°“ *mteal process plants, com- 
terns such as power p * d under sea vehicles. A 

nrand and control systems, ' e severa i human opera- 

supervisory eonltol sys cit essentially the same. 

lots, bn, the general ^ „„ iake , c d„„- 

M„i t , p ie opevatots ay P^ ^ # hybliJ of thc old and 
dancy, or becaus general manager. 

the new control structu 8 contro i system include 

Characteristics of a feeing exer . 

autonomous op rat d proce ssing of information 

° 1Z 1 fro tie supervisor, and indirect but high-order con- 

;:r2t IS goal- or effect-onented. That is, supervisor com- 



mands are interpreted into sets of soecif.V- ™ 

troi inputs to subsystems at the task /eve ’ ° rdln3 | ed con - 

for information presentation -mt , ' Th<? mechanism s 

purpose displays and controls suchrCRT f PUtS "* mUlti ‘ 
legend and function c U ,i.,i , S Programmable- 

tracking device" *"* *•»«*. and 

freer, r„:r,: f e't,d‘t sof subsys,em f "“™ — * 

entire system “ "e, a °«' 

tile operator ar I ™ °. f “• Tire task* of 
monitor overaJ] function t0 Set initlal condit ions, 

necessary. The supervisor att * ! justments > an d intervene as 
goals instead of the implement V° 3Chleving ma J or system 
of the human Ja^Z^ST "“t™' ^ ^ 
deal with the unusual or u£^^” t ^ nce 
event of an out-of-tolcran^ P d ' Intervenin g in the 
-per,is,,,- S p r , maryres ;:“°;;;"''W , s the 


Contrast of Older and Newer Systems 


Older and newer systems differ in the types of taste t • u 
humans must perform and in the behaving Ch 

are called upon to perform these tasks Old? pr0Cesses that 
assimilation of discrete h , , s ' ® der systems require 

system and discre Zim f .T 'T'"' 1 f " nC,io " s » f 
are physic, ™ 'o “ d WaT LT" ^ ^ 

and do are largely man , ^ ° perator know 

determined by the physicllT " pr ° Cedurea which are 
the characteristics of the f 7 °* the Subs ystems and 

mechanism. dlSpi3y lnStru '" e ™ and control 

Newer systems are information-processing oHp ♦ a 
Supervisors are several ® oriented. 

level of functions w f '° m tbe sllbs >'s«em 

and selected data of m abs,r>cled ' summarized. 

end. t2T£r ‘T ««- 

information make derki P 7 ^ t3Sks are t0 inter Pret 
■rise. ,T ^ Probl “" s » h “ ^ 

aclions, an/^r— * ~ !'*”’«* “ d 
judgment. r ratl °nal thinking and 


Implications for Applied Behavioral Research 

thanTsimple IT"*’' 5 ' *«*" iS ™°" 

ntalion frequency changes thf 8 au,omation - Although auto- 
no, necessfrily LutT 7 s “ p ™°''s workload, it does 

so tfi^rs, f> 0 ,~ 
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^““^--^recoumoselhe 

•r=2S “* 

authority and ° f 

control system At th* n t0 the computer- 

how to afford the supcrviLTccesTto’ mT qUeSti0ns of 
control at the subsystem level if j t beers' ° n ^ aU ° W 

how to portrav this inf ” betomes necessary, and 

r Portray this information and effect contml n, 
kinds of questions „,„n „ i - 1 contro ‘- These 

the nature of applied behavioral 

£“777 d,,wcd m, » 

or a r::L°:;::r„rst;;r n 7 k,,,d - ~ « 

on how to design thp i a viorai concerns center 

errors and max , X sS a a , „7" d '» 

performance. P dnd 3CCUracy of the gator's 

are the noUons^ha^tte^^^s'J ^ PC ° Ple ^ d ° 
options to do something different PitT ^ ^ 3CCeptable 
equipment constraints limit these choices' filETh?" 18 “ 
tors will simply try to do the best they c ° PeCa ‘ 

criterion of what is tr, h„ a ^ , ney tan > or there is a 

thereisaruleforthebehavior 0 " 6 ' n ° ti0nS lmply that 

immediatT c^tndTnvoIvll^nttth thf 0 !, 1135 ' reaSonab]y 
characteristics of the subsystems ^ SU systems ’ and th e 
fi e., they have fecd n ^ , eSSe,Ui3,ly immutable 

wired), the behavioral i " 1 ^ 1 Pr ° Perties or a ^ hard- 
interaction in an older T* dssoclated with man-machine 
Concern Zs“ „ L" "’ a ' C SUP ' !rflCia, “ »me degree, 
rnstrument and Ihe acual^el, "" ^ 

by these dcvfces f and d he C abT| 1,J | syslems is S ove rned 
manner. It haf ta,Z o Z * ,h ° m *" tbe P'eacribed 
goal or exlcrnal interactions of o" ° W tl th "" ' ysl "u 
The behaviors of ta e fh °. f S “ bSyS ""’ W " h aMtl “'- 
W interpret an instrument r ma ‘ nly lhose "" d ' d 

control (motor-controf sDcef| IS10 " PCrC ' p,i »"T »P«»le a 

linking ^ 

response relationships. 1 systems ,s stimulus- 

tnatafT^rinfemrlT" 1 '“ kS i "" E^a,i0,, of “T”- 

always been a concern s" ° f 3 general g° a] have 

quently need to be inm , ° f lnformat ‘on that fre- 

° r t ncorp °rate d in a 

S'anTrSl'TnX:, 8 ’"*' 8 S '°" P ' d anfle 


of divided attention, vigilance, and compatibility of delays 
and controls in an effort to improve the human facto.s 

aspects of system operation. 

Providing new capabilities, reducing personnel require 
ments and reducing human error are incentives tor improv- 
ing a system. Solutions to human-related problems in systems 
tavc been directed toward automation. This has been 
approached largely on a piecemeal basis rather than as a part 
of an integrated development ot the entire system. 

The piecemeal approach has had adverse consequences for 
operators, particularly the crew of aircraft (Wiener and 
Curry. 1980; National Research Council, 198. •.Sheridan an 
Hart 1984). Automation does not necessarily re lev 
on orator of workload, but simply may shift the burden from 
physical activity to mental activity. The increased use of 
automation, however, is a trend toward fully realized super- 
visory control systems. The issues are no longer wheth o 
not to automate, but how to integrate automation to 
the human effectively act as a system supervisor. 

The relevant applied behavioral issues extend much 
further into the interaction of the human with the system. 
Human supervisors arc removed from the subsystem level, 
and have a broad span of authority and responsibility . Aut 
rnation of routine manipulative- and continuous-control 
tasks, as well as those for which there are fixed procedures or 
rules, leaves the human to perform decision making an 
problem solving tasks. If more than one human is involved 
in the system, the supervisor’s task will be to manage the 
human as well as the machine resources. 

The supervisor will operate at the system goal level and, 
in effect will be interacting with several levels of system 
function. Since routine problems arc handled automatically, 
the difficult problems, those for which there are no predeter- 
mined rules, must be handled by the supervisor. This will 
require a greater in-depth knowledge of the system functions 
by the supervisor than is now necessary tor the human oper- 
ator Predominant tasks of the supervisor will be knowledge- 
based rather than rule-based or skill-based (Rasmussen, 

The behavioral factors of most concern will be informa- 
tion seeking, assimilation, integration and interpretation 
decision making, problem solving, and resource management. 
The man-machine interface problem will involve integrating 
deeper levels of the system with higher-order cognitive pro- 
cesses The research challenge will be to discover a means to 
achieve cognitive compatibility between an automated con- 
troller and the human supervisor to allow the supervisor 
monitor the status of the system in terms of functional pur- 
pose and to convey goal-oriented commands to the system. 
As Singleton (1976) has written. “The whole point of man as 
» «, .p-d .0 OP operator is rlia. he nee s .0 I nr 

able to make intelligent responses which m turn imp es 
he reacts in terms of concepts and not in terms of stimulus- 

response units. 


In general, applied behavioral research issues in the newer 
systems will shift in emphasis from what operators can do to 
what supervisors will do - or. for existing systems, what they 

d ° In' contrast to older systems, supervisors may have several 
options available to deal with a problem or achieve a " 

The criteria for their behavior are not as clear cut, except 
terms of the outcome. Also, the critical aspects of their 
behavior, information processing, and decision making will 
not be as accessible for measurement as a control manipula- 
tion. None of these changes will make it easier foi u 
applied behavioral researcher. 

F In audition to the shift in the behavioral processes of 
interest there is a trend to expand investigation of the scope 
of variables that influence behavior beyond the immediate 
situational or equipment characteristics. Perrow ( 1983) i ha 
pointed out that the organizational context (i.e.. soc 
structure of the work environment, including management 
attitudes, peer pressure, and personal goals) influences t te 
decisions people make in the operation ot a system. As an 
example, Woods (see appendix E) stated that nuclear power 
plant operators take many more positive actions in a raining 
simulator than they do in actual operations. He attributes 
this difference to the reward and penalty structure operating 
in the real work environment that biases operators to not 
take any action if there is uncertainty about the need or 
outcome. 


RESEARCH VEHICLES 

A lesearch vehicle is simply the facility used to support 
the research. Research vehicles can be classified into three 
categories: the real world, simulation of the real world with 
varying degrees of complexity, and the laboratory. Each has 
advantages and limitations. 


Real World 

Except for research on new systems, the real world is the 
beginning and the end stage for research. The real world is 
appropriate for exploratory-studies to determine the factors 
associated with a practical problem, and it is the fina proving 
ground for evaluation of new developments studied in sunu- 
lation or in the laboratory. 

F ,om the viewpoint of the customer interested in practi- 
cal applications, it is the ideal vehicle for conducting a study . 
It is the onlv context where all relevant factors opera . 
Organizational goals and requirements, stress from real 
hazards, peer pressure, self-esteem, and long periods of rou- 
tine activity are among the most important conditions that 
infh ence behavior. They occur naturally in the real world. 
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“1 T ;" y — ”> research e„y„„„. 

•nents The real world is the best choice for behavioral 

research on what people will do in contrast to what they 
tan do. J 

The real world, however, does have several disadvantages 
for research purposes. There rarely is an opportunity to con- 
trol all conditions; rare events are almost impossible fo study 
hazardous situations cannot be duplicated; experienced sub- 
ject are necessary for a system of any complexity; and data 
ex, ec ion is difficult and can be very expensive, particulady 
it systems such as an aircraft are required. 

Simulation 


Simulation includes a broad category of research vehicles, 
ranges from comprehensive representation of the opera- 
tional equipment and environment in support of full-mission 
P rformance to limited simulations involving a single item of 
equipment or a short task. The advantage of simulation is 
combining real-world hardware, most environmental condi- 
tions, and task demands with the ability to control events 
n conditions. Rare and hazardous events can be intro- 
duced. any required data can be collected, new equipment or 
procedures can be incorporated with the old, and usually 
data can be collected in a shorter period of time and more 
economically than in the real world. 

Limited simulation is particularly useful for the testing of 
concepts or theory derived from basic laboratory research 
or adopted from contexts unrelated to the one of interest’ 
t also is well suited for investigation of problems that have 
been isolated by exploratory research in the real world or in 

The disadvantages of simulators as research vehicles are 
generally proportional to their complexity. The disadvan- 
tages include the effort and difficulty required for setting up 
and controlling the simulation process, and the latitude of 

This 3 ! Sm C °H UrSCS <,f , aCU ° n 3Vailable t0 the test Ejects. 

and the f C CndS l ° mdkC CaCh fUn Umque in some aspects 
and. therefore, can create data analysis problems. 

Either experienced or trained subjects will be required for 

simulation-based research that involves anything more than 

difficult 1 T ThC SUbjCCt training Problem becomes 
ficult and takes on another dimension if the research 

a dresses new equipment or procedures and requires the per- 

1“ °. f COmp t X taSkS approachi "g a full-mission con- 
text (Sheridan and Hennessy, 1984). By definition, there are 

no experienced subjects for new developments; training of 
naive people can require days to accomplish, and can be very 
costly. Ff personnel experienced in an existing system with 
omc similarity to the new configuration are used as subjects, 
iey also will require a certain amount of training. With 
experienced subjects, there is great concern that old habits or 
preferences will influence their behavior, and there may be a 
negative transfer of learning. 


Measurement is a problem in complex simulation studies 
Although the type and amount of data that can be collected 
arc large, considerable effort is required for the development 
measures. Complex simulation is used for behavioral 
research only when high-order, complex, or subtle behavior is 

th e X- an f d f rCqUires detai,ed examination. For example. 

effects of fatigue on performance, decision making in 
unexpected or emergency situations, and supervisory control 
o automated systems are typical applied behavioral studie 
that could require complex FMS as a research vehicle 
Because of the nature of these problems, performance must 
measured in terms other than simple, overt acts or psycho- 

zz:r n,ca " inef " 1 " spNu ° r 


Laboratory 

The laboratory is best for initial test or elaboration of 
th oreticai ideas about basic behavioral processes, prelimi- 
iary research on equipment characteristics such as display 
coding and formatting, control methods and. to some degree 
measurement methods. Its main advantage is the ability to 
isolate and study a particular behavioral process. Subject 
training needs usually are minor, and no special expert- 
nce is required. The set-up of equipment is short and easy 
compared to configuring a simulator, and data can be col- 
lectcd qmckly. The disadvantage of laboratory research for 
app hed behavioral studies is its remoteness from the realistic- 
context rom which most practical problems arise, it is mo 

t“^ 


Comment 


he levels of research vehicle described above are really 
points on a continuum. Even the real world and simulation 
overlap when some aspects of both arc present. For example, 
the Total In-Flight Simulator (TIFS) at the Naval Air Devel- 
opment Center is an aircraft with two cockpits, one for 
research on control and displays, and a customary one for 
safety of flight. Moreover, the research cockpit can operate 
ground-based simulator. The military air combat maneu- 

and thrift a 7 an ° ther eXample ' Here WCapons delive ry 
nd their effects are simulated, but the aircraft and air 

defenses operate in the real world. 

Limited simu'afion and laboratory settings also blend 
gether. New systems frequently consist of electronic dis- 

Sa/Zw» CR J S> COm ' 0lS ^ com P uter .0 Z 
actual hardware, la a sense, simulator technology has been 

incorporated into teal systems. Since CRTs and compute,” 

become principal lools in the behavioral laboratory 

many features of tea, systems and simulators can be created 
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fidelity and validity of research 

SIMULATORS 

The required characteristics and ^phedTehav- 

vchicle arc prominent | ssueS he tcsearch vehicle of choice, 
ioral research. It a sinm a t,r f fide i it y require- 

t'-e — , wm ; and the closely related con- 
ments. In this sec the determina- 

»" J,SC T(„ TeSlmulalors. 

lion of the requiremen s should determine the 

ch 2: rj-is » « ucd " sM> 

are; f rcs carch required by the problem (i.e.. 

1. The type of rescarcl 4 of the varie ties of 

exploratory, hypothesis testing, 

'Tk-w^c of the factor s that innn.nce the behavioral 

processes of interest. specificity of represen- 

The type of research die J ye Know ledge of the 

tation that the re » earc ' v L fe havlora l processes determine 
factors which influence yehicle must represent an 

how comprehensively t e res situat ional conditions. 

commonly thought of as the 

fidelity of the research vehicle. 


Fidelity 


■ • • Fidelity is a confusing term. Much of the 
Definitions- Fideh V t0 talk in qualitative gener- 

confusion is due to th single-dimensional character- 

alities about fidelity as f j simulation and apphea- 

istic and independent of a p nsus on a definition 

tion. Although there is a ack has generally been 

of fidelity of simulation (Hays, 1WU- 

discussed from two major vie ™ p01 " S ' a physica i character- 
The first viewpoint treats i example ^ Huff and Nagel 
istic of simulator eqllipl ^ el y the objectively measurable 

*• °rr' s,! “ m and 

simulator equipment in formed funOK > ^ ^ Qf feehav . 

The second viewpoint treats ™ 1 y ^ petceptual fide l- 
ioral effects such as psycho g^ e fldeBty (Spears, 1983), or 
ity (Matheney, h 5955 ). As stated by the 

behavioral outcome (Jon “ Group on simulation 

National Research ‘Council^ ^ simu l a tor design 

(Jones et al .1985). C ^ ^ ^ of potential 

can be avoided in terms of physi- 

effectiveness for a planned use ratner 

cal correspondence.” Semple et al. (1981): 

A similar view was expr degree t0 which cue 

“Aircrew Training Device fidelrty ^ ^ and 

is " amed in the dev,M 


will enhance performance of j e ‘”, ,|. (1981). and 

— ! 5" — d - 

consist of real-world equipment driven ^ ^ 

devices that externally appea | f tiona i 0 r devices that 
internally very system but 

have no external resei mathematical model, 

correspond functional y > ftom fidelity by 

Often it is assum effective use of simulation; 

abstraction are detnmental Jf detrimental, particu- 
however, abstractions ^ ^ $uch as cockp ,t pro- 

larly in training. There are s some instruments 

cedures trainers m avratmn is sufficient to 

with pictures because this STEAMER (see 

fulfill the intended train t P 1985 ) for a descrip- 

National Research Counci ( ° n f ' N fri gate repre- 

tion) is a simulated P-^^^or graphics display and 
sented schematical y on plant. The simulation 

driven by a computer model oi the ^ in „ 0 

functionally correspon s o the prapu lsion plant, 

other way a physical KP«“ n utacy . Real-world charac- 
A „„.h e ' fa«‘ o Mehty of preesron In 

teristics may be included ^ an adv antage for 

some instances, reduced pree exa mple. there are cases 

some purposes, such as trainl ^ model 0 f actual flight 
where a full engineering ma \ he ™ t ' C nacc ta ble control task, 
dynamics produced a P efceptu ^ lified or adjusted models 
but the dynamics produced y P fl (NAT0 

were perceived to respond more like 

AGARD, 1980). thematica l model had to be simpli- 

lr, these cases, the matlrema or reducing gams 

fied by removing terms from 4 handling quali- 

„„ some terms (by » much , as » m m(Hllfy . 

" srs u « 

represented. enme real equipment, envi- 

A third facet is comp e e • be iepresen ted in a 

ronmental conditions or agents a y behavior 0 f inter- 

siinulation ' IZ ' d0 b ;° B e. ah (1958, used a cem 

prediction l “ k * hCy 

studied. f resea rch vehicle- It often is 

Fidelity in the f°™ °{ ncver a disadvantage, but there 
assumed that high fidelity nQt always t0 strive 

arc research as well as prac 1 fidelity implies a com- 

for maximum fidelity, n g e " e ’ ' ' ld Tbe problem for 

prehensive ^^uations. many factors 
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establish the characteristics of the simulator required to sup- 
port the research project. It helps to keep the simulator cart 
behind the research horse. 

However, it oversimplifies the issue to say that a simula- 
tion should have all, but no more than, those characteristics 
that directly affect the behavior being examined. This is an 
ideal goal, but one that rarely can be achieved. For reasons 
which have been discussed, the customer frequently will 
require characteristics of the simulation that are stated 
vaguely in terms of fidelity. Although researchers frequently 
will acquiesce to this requirement, they should always eval- 
uate the implications for control of the situation, and the 
variability of performance that might result because of the 
presence of extraneous factors. 

Experienced operational personnel used as test subjects 
will also have expectations about the fidelity of the simula- 
tion. This can be both good and bad. Experienced operators, 
in voicing concerns about the fidelity of simulation, may 
identify factors that affect the behavior of interest in ways 
that the researcher had not conceived. This is obviously 
good. 

It is bad. however, if the subjects develop a negative atti- 
tude because their expectations of fidelity are not met. This 
will inevitably affect their behavior. Fortunately, there are 
other ways to deal with this problem if it is impractical to 
meet the subjects’ expectations of fidelity. The alternatives 
are to educate the subjects about the reasons for the simula- 
tion characteristics, motivate them through instruction, and 
train them in the simulator configuration used. 

In short, the concept of simulation fidelity, although 
intuitively compelling and in widespread use, is difficult to 
quantify. It is equally difficult to determine the level of 
fidelity necessary lor a specific application. 


Validity 

Including all factors believed to influence the behavior of 
interest is no guarantee that the behavior in the simulated 
situation will be identical to that which would be exhibited 
in the real world. It always is desirable to empirically deter- 
mine that equivalent behavior does occur in the real world 
and in simulated contexts. This involves the validity of the 
simulation. It should be established whenever possible. 

Definition of validity— Validity is defined as the statistical 
correlation between two sets of measures collected under dif- 
fering circumstances. Depending on when and where the two 
sets are collected, differing types of validity can be defined 
(McCoy, 1963). The principal concern is predictive validity, 
the degree to which measurements made in simulation corre- 
late with the same measurements made in the real world. If 
the two sets of measures agree, the simulation is considered 
valid for the conditions under which the measures were 
made. 


Establishing the validity of a simulator is recognized as an 
essential requirement. Recently, a committee of the National 
Research Council was asked to assess means for improving 
the value of the Computer Aided Operations Research Facil- 
ity (CAORF) as a research tool. CAORF is essentially a ship 
bridge simulator used for maritime research (National 
Research Council. 1983). The committee concluded. “The 
single greatest deficiency of CAORF is the lack of validation 
for its uses. Specifically, CAORF’s mathematical ship models 
and data on training and other human performance charac- 
teristics need to be compared to actual ship behavior and 
human performance in the real world (underlined in the 
original). Validated models and studies would be CAORF s 
single contribution to maritime research and development, 
and need to be given top priority.” 

Verification of validity- The concept of validity is well 
defined, and verification is considered essential for simulators 
used for research intended to elicit real-world behavior. 
However, the testing which is implied rarely can be done 
easily, and often cannot be done. 

Frequently, research is done in the simulator because it 
would be too dangerous or too expensive to do in the real 
world, or because the real system docs not exist. Under these 
circumstances, validity testing is either extremely difficult or 
impossible. Practical constraints also minimize opportunities 
to perform validation studies. Applied behavioral research 
that is part of a system development program usually has a 
restrictive schedule and budget. There is rarely time or 
money to collect data “twice” to test validity. Consequently, 
few formal behavioral tests of simulator validity have been 
performed. 

I, : the absence of direct measurement of behavior in real 
and simulated situations to verify the validity of a simulator, 
efforts often are made to establish validity indirectly by veri- 
fying the physical fidelity of the simulator. The rationale is 
that if the simulator is a comprehensive, concrete, precise, 
and accurate representation of an operational system, then 
the behavior produced should be equivalent to the behavior 
prociuccd in the real world. 

This point was made by the National Research Council. 
Working Group on Simulation (Jones et al., 1985): “...pilots 
now may be certified in a simulator that meets rigorous 
fidelity standards established by the Federal Aviation Admin- 
istration. Such a simulator must faithfully duplicate physical 
and functional characteristics of an aircraft as well as the 
conditions of flight. Similarly, in engineering design, where 
critical and expensive design decisions may be based on per- 
formance in a simulator, high fidelity is the best insurance 
foi obtaining valid performance data. For these applications, 
some lesser degree of fidelity may also produce valid perfor- 
mance data but it is usually not worth the cost or the risk to 
make the determinations experimentally.’ 

Matheney (1978), in a discussion of the need for behav- 
ioral fidelity of simulators used for research purposes, intro- 
duced the concept of performance equivalence. Research of 
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the human pilot in vehicle control has advanced to the point 
where good models can be determined for the pilot control 
function (a process called system identification), allowing 
mathematical analysis and prediction for the overall control 
system. He concludes that we may be able to establish the 
behavioral fidelity of a simulator for the control aspects of a 
task through application of system identification procedures. 

Can) (19/7) proposed that flight simulator fidelity be 
assessed using a backward transfer paradigm in which experi- 
enced pilots are tested on their ability to fly the simulator. 
Significant deficiencies in performance would then be taken 
as indicators of simulator fidelity deficiencies. He states, 
"While backward transfer should not be the sole justification 
for simulator procurement, one would be hesitant to use a 
simulatoi which could not be operated by competent pilots.” 
Applied behavioral research is intended to gain new 
knowledge about human performance to answer a practical 
problem question. The "bottom line" is that without specific 
validation studies, or without specific theoretical knowledge 
about what factors influence the behavior of interest and 
how. the researcher is forced toward high fidelity of simula- 
tion to ensure that valid data will result, and to achieve 
acceptance by users of the data outside of the research 
community. 

There is a lesson for research to be gained from experience 
with training simulators. It is evident, when considering 
transfer of training and training effectiveness, that much 
depends on how the training device is used. Although the 
characteristics of the training device (i.e., fidelity) can enable 
good training, it is up to the instructor and automated train- 
ing features to produce a high transfer of training. Similarly, 
high fidelity may be an enabling factor tor a research simu- 
lator, but the manner of use is a prime determinant of the 
validity of the data. More is involved than the physical and 
functional characteristics of the simulator. 


This is especially true for full-mission aviation research 
which requires simulation of many real-world features in 
addition to those of the vehicle 'cockpit, visual system, 
motion platform, and equations of motion. Considerations 
of the fidelity of simulation must include the total environ- 
ment, e.g., ATC, ground facilities, cabin crew, weather, and 
other aircraft. 

Moreover, consideration must be given to establishing the 
framework of regulations, procedures, and preplanning which 
can have a significant effect on human performance. If 
behavior that skilled people exhibit in the real world is to be 
elicited, then the conditions that would exist in the real 
world must be established, including familiar missions and 
procedures. If the research is concerned with crew coordina- 
tion and decision making, it must be recognized that the 
extended team includes ground team members in addition to 
those in the cockpit, along with all of the briefing and plan- 
ning preflight activities. 


CONCLUSION 


All of the factors discussed in this appendix are part of 
the process of determining how a study is to be conducted 
and how the research vehicle is to be used. If the study is to 
be done in a real-world context, these factors should be 
addressed in the processes of study planning and experimen- 
tal and scenario design. The experimental design, procedures, 
and scenario emerge from an iterative consideration of scien- 
tific and practical goals of the research, the available 
resources, and the constraints which are imposed. In this dis- 
cussion, considerable attention was given to the issues of 
simulator fidelity and use of the term "validity” because an 
understanding of these concepts is central to the planning 
and execution of applied behavioral research studies. 
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Pittsburgh, PA 


David Woods has conducted two studies on the design and 
utility of safety parameter display systems (SPDS) for 
nuclear power plant (NPP) operators. The purpose of his 
research, sponsored by the Electric Power Research Institute 
(EPRI). has been to evaluate SPDS design characteristics. 
His approach to this problem was first to determine how NPP 
operators would use an SPDS, e.g., as a diagnostic aid, as a 
planning aid, or as a control aid. That is, for what tasks and 
circumstances would an SPDS be an improvement over the 
conventional control room instrumentation? 

For his research he used an NPP training simulator with 
the trainees, who were experienced operators, as subjects. 
The NPP simulator used is heavily scheduled six days a wee 
The experimental work was piggy-backed on the tr ^ in 8 
exercises. This can be a very cost-effective means tor MOR, it 
the experimental goals can be adapted to the on S^ing 
training situation. In the present case this was accomplished 
by using test events devised by Dave Woods instead of some 
of the training events. The trainees had no way of knowing 
the difference except for the fact that several additional 
observers appeared at odd times and for about half of the 
events the SPDS was turned on - a giveaway that an experi- 
mental trial was being conducted. The instructors were 
cooperative in allowing the experimental events to be used 
because they were very similar to the training events (thus 
fulfilling both the training and the research objectives) an 
because there was no grading of the trainees involved. 

Nuclear power plant operators are required to undergo a 
week of training every year. The training consists oi a mix o 
classroom work and simulator practicums. These are essen- 
tially problems involving a variety of emergency and out-ot- 
tolerance conditions. The trainees are not required to pass a 
test or meet some performance standard. The requirement 
is simply to undergo the training. At the end of the simulator 
exercises, the instructor does debrief the trainees on their 

performance. r 

One of the principal problems in evaluating the utility o 

SPDS to NPP operators is gaining access to their perfor- 
mance. Their tasks are primarily monitoring and decision 
making. The overt behaviors associated with these tasks are 
frequently few and furtive. Information for commonly 
occurring or familiar simulations can be assimilated rapidly 


md a decision about what to do, if anything, arrived at 

mickly. . . , 

Dave Woods used a clever method to reveal more about 

he information seeking and decision processes and also to 
improve the tractability of measuring performance. Several 
3 f the experimental events were designed to be slowly emerg- 
ing situations. That is, even after the operator detected 
abnormal readings on some indicator , it took quite a while to 
get other information and formulate a tentative hypothesis 
of what was causing the problem and what to do about it 
Even after some actions are taken, significant time is required 
to gain feedback information to confirm that the problem 
has been properly identified and the correct actions 
pcrfori icd. 

In addition to using slowly evolving events to reveal more 
about the behavior of interest, Dave Woods constructed 
events that superficially appeared to be a common failure to 
mask an unusual failure. This was most easily achieved by 
devising multiple-failure events. For example, a scenario 
event would include the failure of an instrument that, 1 
operating, would signify that the event was an unusual and 
serious problem. The failed “leading indicator,” however, led 
the operators to believe a minor, common problem was 
occurring, thus complicating and stretching out the process 
of identifying the problem. What would otherwise be a 2- or 
3. min exercise would now run from 10-20 min or more. 
Choosing scenario events of this sort is a device for expand- 
ing the information seeking and decision time so that many 
more behavioral activities can be observed in detail, t is 
dearlv a technique applicable to aircraft simulation research. 

Subject matter experts (experienced NPP operators) were 
used to develop the experimental event scenarios, as well as 
to collect observational data during the tests and to interpret 
and evaluate the performance data. It usually took several 
iterations of refinement to get a satisfactory event scenario 
For 'he two studies performed by Dave Woods, 12-event 
scenarios initially were developed. After preliminary testing 
by several groups of operators, he settled on seven events for 
the formal experimentation. 

The primary data for the experiment were descriptions ot 
the actions of the operators by other experienced operators 
Dave Woods said it would be very difficult to obtain usefu 
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automatically recorded data on the type of behavior involved 
in process control. In effect, the performance measurement 
system would have to be almost as smart as a human expert. 

n this case it was a lot cheaper to use a human expert. The 
automatically recorded data that were collected were in the 
form of a time record of the state of the NPP. The experi- 
mental events were also video taped for use during the per- 
formance assessment stage. Dave Woods refers to observa- 
tional data collection as the ethnological approach to perfor- 
mance measurement, known as “watching the wolves mate.” 
During the first experiment there was no information on 
how the operators would react to the test events. After some 
experience of observing the responses of the operator, it was 
possible to develop a rating sheet for the SME observers to 
use in the second study. The rating form allowed for the fact 
that not all teams of operators would follow the same steps 
in dealing with the event. It included the full range of poten- 
tial operator behavior. The observational and recorded data 
were collected and then assessed by other individuals. Dave 
Woods believes it is important to separate the collection of 
observational data from its assessment to minimize the loss 

of information from interpretive filtering at the collection 
phase. 

EPRI originally suggested that observing the performance 
o operators in a high-fidelity simulation with the SPDS 
available, or not available, along with a few simple measures, 
would be sufficient to evaluate the SPDS. This, of course, is 
the common attitude of sponsors who assume that studies 
involving substantial behavioral components can be per- 
formed in a manner analogous to a shake-down test of equip- 
ment functions. A by-product of this expectation on the 
sponsor’s part was that Dave Woods had considerable latitude 
in the approach taken in the first study. In effect, it was an 
exploratory investigation that served several purposes Candi- 
date experimental events could be tried out. The range of 
responses by the operators to the event scenarios was discov- 
ered as well as their apparent difficulty. Also, interesting 
leads could be picked out for more focused and more effi- 
cient investigation in the subsequent study. The first study 
also served to educate the researcher about how to conduct 
experiments in the NPP training simulator context. One of 
the most valuable lessons was on measuring performance 
The first experiment was fairly vague in purpose' and 
simple in form. Dave Woods talked to several experienced 
operators to gain some ideas about how the SPDS might be 
useful. The events chosen for the experiment were typical of 
events used in other studies and thus could serve as a refer- 
ence point to tie the results to other work. The operators 
were trained for only about 2 hr on the use of the SPDS 
prior to the study, so if anything the results would be biased 
for the conventional instrumentation conditions. 

The conflict between the desires for face validity and 
experimental control in NPP simulator experiments is no dif- 
ferent from any other simulator-based research. Dave Woods 
beheves that any compromise should be in the direction of 


gaming better control. He is willing to sacrifice some realism 
if the result is to drive or nudge the operators to the situation 
where the performance of interest occurs. 

If the purpose of the experimentation is to derive infor- 
mation that is generalizable to other situations, there must be 
some concept that a condition in the simulated context was 
chosen to represent. A restricted or reduced simulation per- 
haps with less physical fidelity, may have greater concept 
validity than an FMS. The reason for this, of course, is that a 
reduced simulation is designed to include only those charac- 
teristics that are essential to the concept. That is, the level of 
abstraction of the simulation approximates that of the 
concept. 

Concept validity is very important in behavioral research; 
therefore less than FMS would be desirable for behavioral 
research in cases where the concept validity is improved by 
reducing the simulation scope. Although high fidelity FMS 
may have low concept validity, concepts must be finally 
tested in this context to prove they are robust and valid 
under near real-world circumstances. Most sponsors will 
expect research with some practical implications or impor- 
tance to use FMS. 

It is often very hard to convince sponsors or research 
result users that FMS is not necessarily the best way to do 
the research. Unfortunately, they are unlikely to get face- 
validity and convincing data out of a reduced research con- 
text. Dave Woods also stressed that it is important to be cer- 
tain that the sponsor or user community has an answerable 
question. Frequently, they think an issue is clear-cut (e.g„ i s 
an SPDS useful); but, from an experimental point of view! it 
is only a vague notion. It is usually necessary to refine the 
question to some tractable form. 

Realism in NPP simulators in terms of user acceptance is 
not as serious a concern as in many other forms of simula- 
tion. The training simulator used by Dave Woods is a high- 
fidelity physical and functional representation of an NPP 
control room. Any issue of fidelity or realism would occur 
only if the simulator did not respond as a real NPP does or if 
the scenario events did not have a plausible basis in reality 
As Dave Woods pointed out, most experienced operators 
have never been involved in a real abnormal event and. 
because of the large number of possible failures, the opera- 
tors have no idea before or during the course of an event 
whether or not it is realistic. After the exercise is over and 
the precipitating cause of the event is explained, then and 
only then are the operators able to comment on the realism 
of the event and its chain of effects. This is obviously of 
great benefit to the experimenter. The researcher has wide 
latitude for composing event scenarios to serve the experi- 
mental purpose. Incidentally, the operators become very 
involved in the scenarios and at the end are very interested to 
find out what the “cause” of a simulated event was. Dave 
Woods pointed out an outstanding issue about use of reduced 
fidelity simulation for NPP research; no one knows what the 
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attitude of experienced NPP operators would be to reduce 
fidelity simulation because it has not been tried. 

Similarly, the inability to assess what is real could be the 
case for pilots of future commercial aircraft. New generation 
aircraft and NPP are both examples of supervisory control 
systems in which ongoing processes are automatically con- 
trolled. The operator only monitors the process and usually 
only intervenes under abnormal circumstances. Like the NPP 
operators, there is no way that a pilot can know all of the 
possible ways the automated systems in future aircraft might 
fail, or know all of the possible indications and consequences 
of its failures. 

There was one realism problem in the experimental event 
scenarios used on the training simulator. All desired effects 
such as failures have to be scheduled by time, and not by 
contingency on some specified set of conditions. One event 
involved the failure of a valve. In the real world it is plausible 
for a valve to fail under pressure or when it is being opened 
or closed; it is not realistic for it to fail while in some benign, 
quiescent state. Yet. in the simulation, because the valve 
failure had to be scheduled by time, it would appear to the 
operators that the failure was capricious if there were no cir- 
cumstances at the time of failure which would plausibly be 


associated with the failure. This obviously detracts from the 
realism of the simulation. 

Valve failure on time alone also had a detrimental conse- 
quence tor the experimental control. Because the valve fail- 
ure occurred on a time basis and was unrelated to whatever 
the plant or the operators were doing, it would happen at a 
different point in the chain of actions for each group of 
operators. Thus the conditions were not exactly comparable 
from one group to the next. It is a relatively small point, but 
any research simulator should have the capability to permit 
scheduling of experimental events contingent on circum- 
stances and subject actions rather than time alone. 

Dave Woods believes that there are some marked differ- 
ences in the behavior of NPP operators in simulators and real 
plants. In simulators, operators arc much more action 
oriented, that is, they are more willing to take some positive 
step in an abnormal situation in the simulator than in the 
real world. He attributes this difference to the fact that in 
the real world operators are very concerned about repercus- 
sions from any actions they may take. The same sort of stress 
does no arise in training simulator exercises. However, he 
did say that there appear to be no obvious differences 
between actions taken in the two settings. 
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appendix f 


FIELD INTERVIEW 
THE BOEING COMPANY 


Thomas C. Way 

Avionics, Crew Systems Department 
The Boeing Co. 

Seattle, Washington 


Dr. Richard E. Edwards 

Managing Consultant. Human Performance Analysis 
The Consulting Division, Boeing Computer Services o. 


Thomas C. Way and Richard E. Edwards have been 
involved in the development of simuhtor -arios for ev^ 

evl!uatils CO otle e B U Sg-proposed version of the C-14 
intended to replace the C-130. The principal issue in the eval- 
uation was workload since the C-14 concept featured a two- 
person cockpit crew instead of four persons as m the C-130 
(nil ot copilot flight engineer and navigator). Tom notea 
that the C-14 concept development was the first ma J or P r0 ' 
IL L included human factors members as part of the 
design team rather than as part of logistics support. 

Richard Edwards until recently worked in the crew sys 
tcnfs dWisions and has conducted a number of simulator- 
based developmental and FAA certification evaluations for 
military and commercial aircraft and systems, including the 

B 7 Bocing 7 |ias three types of aircraft simulator facilities^ 

1) flight crew training; 2) developmental, which i range 
quality from rudimentary to near flight-training i^.and 
3) engineering, which are readily reconfigurable but have 

rCl A^l'ysirtilnurator-based performance tests, and ques- 
tionnates ,v,„ .0 educed pilou are ™ 

wavs of evaluating new equipment or systems. Hiese 
methods are applied extensively for workload determma^ 
tions. It is one of the principal concerns in commercial and 

military aircraft development. . _ 

The c-14 workload evaluation started with an analysis o 
imposed workload (task load) for various situations The 
focus was on the crew's ability to perform navigational tasks 
as well as piloting tasks. Eliminating the flight engineer was 
never considered^ to be a problem. The ™ 

scenarios were 35-45 min segments, and > e “ 
seated was in the vicinity of the Rein-Main AFB Germany 
Ten C-130 crews were used for the evaluation; each crew was 
available for only 2 days. The first day was devoted « train- 
ing, primarily on the use of a control and display unit (CDU) 
for navigation, since its use would be a critical factor in the 


simulation evaluations. The second day was devoted to 

“The simulator was a flxed-base cockpit with a mono- 
chrome projection visual display on a 1 ^ “ v' for takcoff and 

ZZXSZZ' ^„r, wuu.d cnic. wca.1,.,^ « 

A„ r „f the flight would be on instruments. A tixcci 

r “ as 1 cargo is discharged. The proved 
system would automatically compensate tor these ^t pes o 
chan°’s so the extraction task was not included in the see 
nario* Aircraft handling was not considered to beanisue^ 
Ai S o a Boeing test pilot working on the C-14 project soul a 
w P „u,d do jus, fine since U,c mono. busc 

in ,t,c simulation. Mos, of the cockpit mstruinents whtch 
were of conventional electromechanical design (other than 
the navigational CDC1), were functional. 

The scenarios centered on creating navigation problems^ 
ThI crews were required to accept in-flight diversions and do 
such tasks as estimate fuel states and insert way-points using 
the navigation GDI). The object of the study was not to conv 
pare alternatives but to confirm that the crew size and 
rnc i nit-desiizn concepts were practical. 

The operational question was whether the crews could 
perform the tasks without feeling excessively burdcned , 
modified Cooper-Harper rating was used to asscss work oad. 
The test scenario would be frozen at a natural task b 
point and the crew asked to rate the segment just complete^ 
T natural breakpoint was considered to occur when the 
diversion or similar problem requirements had been com- 
pleted and the aircraft was beginning a cruise segmen . 
considered more desirable to disrupt the continuity of the 
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scenario to get immediate ratings than in r( aU, 

after the scenario was completed ' °" mem ° r - V 

and T exLuTe7vCq^Wy , aL7fC ^ phnned 

r , r « reformul ; te fr':^ p r zztz;:: 

'° ln . flUe K n i ce thc design proposal. He satd 

sj 7 rr ? t “ 

tlic overall mission workload n 

At 11, Is point. Ton, Way was callcd 

r h 0 ,l before he had an 


™^n U,tls 0F emulation FOR 
RESEARCH AND EVALUATION 

injt 7|! n | 3 "7 ful !' mission Emulator is installed, there is 
> ow emand for its use; consequently everything is 

, 1 17 ,:r T r even " ,oui! " ■ ^ - KX 

dy may have been more appropriate. Later when the 
i" ?“ V" tlCa,> ; demand - 1» *ue Of studies 
conct, *' »«««» one of 

There are four principal reasons for doing FMS: 

1 • To resolve a collection of related problems If there k 
» fenes of par,-, ask evaluations c ,„ ed “7a« “ ' “ 

»mTohl'',i V ' ,i "' i ° nS ° f S ' ,C,aI diff ' ,Cn ' and 

:::x:;" rgendes as a 

in te res f ResuiK te f aCti0nS ™ y in ' ,Uence the behavior of 
HU realistic, or if the scenario is too short 

fo„„a,s change during ,he phases of a descen, ,o landing P ? 


However, sometimes a simulation requirement can 

“■ 7 ** “xrrx: 

single simulator cockpit. Problems with the individual com 

(MTBF) ?? Un ° Veral1 Mea " Time Between Failure 
(MTBF) of less than 1 hr. They eventually gave up the Z 

2 rllT' m, ‘ eMcd c, " M,ta > ° f u “ - « 

tionZwhen ZT^ “ 2PPr0priate for search or evalua- 

influenced bv se 7 rCaS ° n l ° SUspect behavi °r will be 
iuenced by secondary contextual circumstances Gener 

a%, studies intended to determine what the best Z7 

oTnot i a „'he bC t n f PCCifk COnditl ° m ’ ° r t0 discover whether 
or not inherent functional problems occur, are the Joe 

domain for part-task simulation. In addition, if Search 
tlC ° n 3 PartiCUlar Pr0blem > do not add extraneous 


SCENARIO DEVELOPMENT 

~re;r 

is 

because the winds always change from day to night. 

e Procedure for scenario development should start with 

variables Tii^ . ^ ^ experimental 

"h„ is \ T , Sh ° ,,ld av “ d ""V effects 

lha, is, a test event should not occur too soon after a ore 

P.ob J inti™ hke'Stere ‘Il7b“ perTo dTstble 

o'K^^r ne ,he ,ppropria,e 
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CONTROL OF SCENARIO 


_ bc dcs igncd so that the subject s behavior 

Scenarios can bt d fc ^ aUowcd considerable latitude 
is either tig i > c study. In most evaluations, 

adherence to a dcs. Station is «ll defined. Mo,e 
because the intcnl exploring fut problems or 

f3 1n mdustry applications of ^“studv « 

“ ^ dc u s ;: ab 0 r c ZZTJS* ** ■* Ukely to be 

possible becaus fewer test subjects or 

^a^h'The^fo re StU every effort is made to preclude deviations 
crews. Therefore ev y and ieduce the perfor- 

from the desired COUrSC ° f u r crews so that differ- 

mance variability among the subjects o C ompli- 

Sis 

^iC' t plough graining in some 

circumstances. involve the experimental task 

Pre. raining Tussle skill required and 

directly but should 1 Training should 

fSmU «d*w ll^dTcm'ibied criteria to minimile perfor- 

zzzzz - «. : 

no, always possrblc when F«> cample, 

it is feasible for proc, idnra a nd k I yp fo , 
a study was conducted l " hjnd conuo |s 
locating certain buttons and switch^ ^ werc uscd 

for an aerial refueling oo . simulator test involved 

« - -*r; wS 8 r"t^ » 

“ kinE j m mlln monie To minimire inherent skill differ- 
grammed in flight pro in terferencc effects among the 

ences and dissipate pro ora ctice before each trial 

configurations, the su F J W P task was touching 

with another set of controls. The pract oard Jhe 

the boom tip to designate ^ajes ^ pcr form this 

practice would continue -un J accuracy. Only then 

would^h^y Writer simulator to perform the experimental 

task. 

DATA COLLECTION 

possible. It results collection and transcription 

to use mult, pie tedundan, 


data recording fo. particularly critical data mh.i rs. twtmlrgi^ 

- “* *?*" ‘I S » fu» time/cvcnt bis- 

l^gs and system states ate fogged and 

Cha A^ S ^taarc cofleTted"^^ forms. Video recording 
is used whenever 

audio and video recordings wer diffcren ces from the 

r,e^:::~oty and prepare fhem •» 

tions or ,’^ ak a ^jgotion^re Afferent ^^sf^ l * ie 

forms o. data concc time t o perform 

mtjrs “Ttin aTudy comparing 

because of concerns a ratine scales and debrief- 

However, pos, flight quest J^Xein with ptefet- 

l n „ 8 ec m ,‘ iiri totTlcslpreferted condition will sometime, 

result ,n better objecfive periotmancm a „ a|ysis , ou . 

" measurement r.rfuires a good 
'“’j S “ X e d bpmen, and , men, ions are frequently 
- - mcmS Clicking of these roniines must be itU.ns.vc; 
Z'y surprises are usually 

nary subjects from the population ot interest. 


attitudes of participants 


Pilots, both military ** 

shortcomings in simulators as fa as tJJ conccrned 

lics , visual scenes an a»r c long experience with 

play formatting and olhcr ch rac er < is , 

‘'“plrsTto ™'T “ CCp,l Xr!ll thenHitter- 

- -<*», — r dinent,:: r«r: i.. 
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view participating in simulations as secondary to their nri 
mary duties of flight testing. Also, the simulat o s t dies are 
often scheduled at nicht and th^ir • ■ e 

unpaid overtime. P«rttc,pai,„„ amounts ,o 

The pilots are always asked to comment on tire Simula 
don and scenario to find out if they feel the sim„l,tt 
appropriate to the purpose. Sometimes the structuring^™ 
seenatto can go too f„. one ease, , scrip, f„, .Zita to 
ad as a passenger briefing was provided. Many pilois fell 
1 s was a bit much, particularly since most saiH 
exactiy what they would say in an annouZZZ ^ 


education and responsibilities of the 

SIMULATOR RESEARCHER 


the!" “^r la ' 0 '- based ™d research 

mere both an experimenter learning curve and an exneri 

memcr reaching curve. The learning involves 

appreetarton of ,,„w difficult I, Is p, a „ and ”, 

S'citr and u"° W ' ne,ia "“ S rou must be 

the eft 01 1 occurs before and after the actua] data 

eTflri !s°i ' I ' C ” Whil£ rUnnin§ SUbjeCtS ‘ Ab0Ut 60% of be 
effort is n preparation, 30% in data analysis and interprets 

.on. Only about 10% of the effort is the execut oi of le 

simulation runs. There is no apparent procedure for deter 

mining how difficult a particular study will be 

ers inUrfinrtoT^ “ "T *”** 1 b >' experienced research- 
imparting to less experienced or novice researchers the 

ZZ n S “ be rnl».i»e intlnee 

executing sZuIal “ P “' S ° f p,ep " i "* for and 

executing simulator experiments and evaluations Every 

large-scale simulation facility should have a cadre of exn"f 

enced researchers to assist in the conduct of studies by cl 

f ; con - 

ZZZ “fc' ' ha ' are . not fullp appreciated 5 by Se 
continued c„„s„,,"aL 

shakedown testing, verification of the XtenSIVe 

performance measurement system, and the atadJte ’n'ecesshy 
for comprehensive preliminary testing. necessity 

Probably one of the most difficult problems of the first 
-me experimenter is becoming familiar with the phy^l 
design and operation of the simulator and the role of the 

beetle the P e° rt PerS ° nneL ^ iS es P eciaI1 >' important 
because he experimenter may have expectations that these 

personnel understand the intent of the study what if im r 
one but the experimenter is going to worry 


be^lnted 15 ’ " U ' S ' b ' "a-« 

h.p A pe"n“ C fi,e ^ " “ 8 “ d what can 

mary preparations were made and the display Ld 
aofbware rnstalled in the simuiaior. 

Sh ti r° servatlona I data and the opinions of the pilots 
Shortly after the test began, the computer locked up this 

w at the same time - « 
nights lookinffnr 1 Spent ’ 0,1 several success ive 

the most likelt P ro S™lng error that seemed to be 

the most likely cause of the lock-up. Of course, much valua 

resource's " 35 ^ ^ WaS 3 loss of the ^arce pilot 

was t aZeT ntUally d ' SC ° Vered that the cause of the failures 
was a tape-drive write-error. The simulator had a built-in data 

t~r Ut 11 WaS n0t ^ USCd ^ particular 
xperiment. The support personnel knew that a tape was 

supposed to be mounted onto the drive to log the Jala bu 
his particular experimenter did not provide oSe cont^y to 
, r "’ a . practlce a nd the expectations of the support person 
nel. Therefore one of the facility people put a discarded tape 
n the drive so the simulator would run The researcher u,a 
-aware of all this, and the well-intention^ sipp" p™ 

oui "° f 16 tapC W3S the S ° Urce ° f the probIe,ai It turned 

, course, that the tape was discarded because it was 
bad, and when it was used on the drive, a write-error would 

occur at some random point. write error would 

The researcher is equivalent to a general building contrac- 

bilities and' iLT'd ^ Simuiator °PC'ation and capa- 
- d the facility management and support personnel 

The researcher must take all responsibilities for the planZ 

» d cheek, „g g„d vetiryipg of hardw.re a „dT,™ 

u-cs pttor to the stody, as well as eopcloeti™ ' 

lection and performing the analyses. 

ATTITUDES OF CUSTOMERS AND USERS OF 
INFORMATION FROM SIMULATOR STUDIES 


The customer’s expectations, whether the customer is an 
-house user or an external organization, have a marked 
influence on the r„„, „f tl „ simnlatoAased e Id 
evaluations. They tend „ w , y s„ cs s face 
requirement, and often expect more to tcsult from the , 
tan is possible. Engineers especially have expectations that 

r r «;„ rr 1 "" sl,oi " d be readi,y 

q tek y acquired to answer their questions. Engineers fre 

they ^ do'To?^ ; XpCrimCntation with demonstration, and 
ully appreciate the importance of good 
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experimental control as a factor in the validity and reliability 
of the data. Accommodating the biases of the customer and 
user to some degree is necessary. It is also possible sometimes 
to explain why certain methods arc important to the goalot 
the test and why certain features of the simulation arc not 
very important. 


PLANNING FOR DEMONSTRATIONS 

Outsiders, particularly, managers, marketing representa- 
tives. senior personnel from the client organization and 
other VIPs do not accept the importance of not interfering 


with the conduct of an experiment. They expect to sec 
demonstrations and think little of entering a cockpit during a 
run Unfortunately there is little that can be done to dis- 
courage this practice. The wisest policy is to recognize that 
demonstration is an important part of any development pro- 
gram and serves legitimate needs even if it is inconvenient 
and sometimes seriously detrimental to the experimental 
plan One of the best means to avoid interference to the 
extent possible is to plan for demonstrations before and alter 
the data collection period, allow for it in time and budget 
plans and be sure key people are made aware of the availa- 
bility for demonstration well in advance. This will not stop 
interference but it will help. 
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appendix g 


SIMULATION 


FIELD INTERVIEW 

STUDIES OF VISUAL ILLUSIONS DURING 


NIGHT APPROACHES 


Dr. Conrad L. Kralt 
Bellevue. Washington 


Dr Kraft retired from the Boeing Company in 1983. 
During his long tenure with the company he performed 
numerous studies of pilot vision and performance, safety, 
and simulator visual characteristics. He received numerous 
awards for his simulator-based research relating illusions of 
altitude during night visual approaches to the visual and 
geographical characteristics of the airport and surrounding 
area His basic finding was that an illusion of excessively 
high altitude was manifested when an airport is situated at 
the edge of a city and there is an upward tilt of the city from 
l°-3° from the airport to the horizon. The pilot of an air- 
craft making a straight-in, night, visual descent from a high 
altitude over water or otherwise dark terrain toward the air- 
port with the tilted city in the background will nusperceive 
the aircraft altitude to be much higher than it is. The conse- 
quence of this illusion is the aircraft will contact the ground 

5-8 miles before the threshold of the runway. 

The research was prompted by a series of crashes of 
B-7^7s within a few years of the introduction of the aircraft 
into" commercial service. Dr. Kraft was asked to determine it 
there were any characteristics of the aircraft that could be 
contributing to the accidents. He discovered through looking 
at the descriptions of accidents involving the B-727 and other 
commercial aircraft, that many of the accidents occurred 
under conditions of approaches at night over dark areas to 

airports near cities with an upward tilt. 

Dr. Kraft was struck by the common circumstances sur- 
rounding this large proportion of accidents and the strong 
suggestion that vision was involved. He began a series o 
investigations on the visual perception of altitude. He began 
by photographing maps of some of the cities where the acci- 
dents occurred. He placed glue on the maps outlining the 
runway and airport and roads of the city plus random spots 
in built-up areas. He then sprinkled fluorescent chalk dust on 
the maps The maps were photographed under black light 
from various distances and angles, corresponding to a long 
descent path to the airport. The set of photographs appeared 
as night scenes of an approach to an airport. Three groups o 
pilots current airline pilots, noncurrent airline pilots, and 
noncurrent small aircraft (low altitude) pilots were asked to 
sort the photographs into altitude bins. The result was the 
first group was reasonably consistent in their judgments but 
the last two groups showed high variability. The lesson from 
this study was that only current airline pilots would be 
appropriate subjects for future studies. 


The next study also involved judgment of altitude based 
on viewing 16-mm moving films of a model city. The camera 
moved down tracks set to represent two high and two low 
approaches. The judgment data from the airline pilots was 
inconsistent. The film resolution was not high enough to give 
an adequate representation of point light sources as seen in 

the real world. 

For the third study, a large model city mounted on a mov- 
in o table was constructed. The budget limit for the equip- 
ment was $12,000. The model was made by puncturing pin 
holes through a large print of an aerial photograph of a city 
and airport. An important detail was to puncture the o es 
with a soft wood backing under the photograph so that eac 
light point was dimpled. This was necessary to preclude t ie 
whole scene from suddenly going black if the simulated air- 
craft went below the plane of the table. The slightly raise 
holes were in effect small spherical light sources. 

Prior to construction of the city model, an analysis was 
made to determine if the model moving toward the pilot in a 
mockcd-up cockpit would provide any monocular cues to its 
true distance. The vision literature indicated that accommo- 
dation. the only available cue because the pilots viewed the 
scene with one eye occluded, would not be effective until 
the table came within 28 in. of the observer. In effect, the 
city rould simulate an approach from 20 miles out to within 
4 5 miles of the runway over altitudes in excess of ..0,000 t 
to below ground level. The city could be inclined up to 3 . 
The runway remained horizontal. The cockpit first used was 
a left-over, single-seat fighter mock-up. The aerodynamic 
model was a nonspecific representation of an aircraft weigh- 
ing approximately 100.000 lb. The only instruments availa- 
ble lo the pilots were airspeed and vertical speed indica- 
tors. The aircraft did not have horizontal, yaw or roll 
movement. 

The Boeing pilots objected to the absence of an altimeter, 
believing they could not make a proper descent and approach 
without it. They were told the object was not to evaluate 
their performance but to determine if the city simulation was 
adequate. This allayed the pilots’ concerns. As it turned out, 
the pilots could fly the descent very well with only the two 
instruments and the visual scene. After the experiment, the 
pilots were all very surprised to find that the city moved 
toward them rather than vice versa. It was a good confirma- 
tion of the absence of extraneous visual cues. 
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Additional funds provided to the project because of a 
need to move the laboratory allowed the simulator to be 
upgraded. A cockpit resembling a transport aircraft was con- 
structed, and the aerodynamic equations were improved to 
represent an aircraft with a gross weight of approximately 
1M)000 lb - somewhere in the weight range between a 
. and a B-727. The pilots had been extremely critical of 
the handling characteristics of the earlier simulator. With the 
improved aerodynamics, there was a marked reduction in the 
variance of the glidepath for each pilot. 

Pilot acceptance of the simulator helped the study 
because word was passed around that it was a good simula- 
tor. This considerably eased the problem of recruiting pilots 
to participate in the study. Another factor which prompted 
cooperation by the pilots was that almost all of them had 
had a close call descending below the glideslope at night. 
The enthusiasm of the pilots about the importance of the 

study also influenced management to continue support of 
the study. 

Because of the limited task requirements, control of air- 
speed and descent rate, Dr. Kraft was concerned that the 
imposed workload was so low compared to actual flight that 
the pilots may be concentrating far more on altitude judg- 
ment than they would be in actual operations. To increase 
workload other air traffic was added to the simulation. That 
is, the pilots saw aircraft beacons flying over the city (the 
beacons were actually small lights mounted on a few rotat- 
ing disks ganged together). Because of the radii of the disks 
and their slow movement they appeared to the pilots to 
move in a straight line. The pilots were asked from time to 
time to report on the azimuth, heading and relative altitude 


° ,, he , °H 1 , er aircraft (v,a a ret ] u cst from an air traffic con- 
troller). This side task both increased workload and drew 
their vision away from the approach task as a normal scan- 
ning ot the airspace would do. The pilots were also asked for 
ieir estimated altitude at precise points in the approach 
This was accomplished by computer control of a tape 
recorder. A nice touch was that the voice on the tape was 
tha of the person playing the role of the air traffic con- 
roller who asked for the information on the other aircraft 
Ihus, it was not apparent to the pilots that a tape recorder 
was making the requests. 

Reflecting on the series of experiments. Dr. Kraft made 
several noteworthy comments. Maximum effort was placed 
on visual-scene fidelity because of the suspected perceptual 
nature of the problem. Fidelity of other features were of 
minimal concern because they were judged to be of little 
consequence to the characteristic of the performance of 
interest - going below the glideslope. The very limited fund- 
ing tor the study forced concentration on the most critical 
feature of the simulation. Had a fuh'-mission simulator been 
made available for the study. Dr. Kraft said they probably 
would have not discovered the basic problem. Conventional 
model-board systems simply could not portray the charac- 
teristics of point light sources and thus cues responsible for 
the illusion would be absent. This study evolved through* 
number of stages deliberately. Dr. Kraft said that pilot 
experiments are really very necessary to help you think 
about a problem. He ventured that anyone who attempts to 
perform an FMS experiment without having conducted 
several preliminary studies of more modest scope would be 
very lucky to have a successful outcome the first time. 
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SIMULATION 


FIELD INTERVIEW 

at the united airlines flight training center 


Dave Shroyer 4 

United Airlines Flight Training Center 
Stapleton International Airport 
Denver, Colorado 


Dave Shroyer has been involved in United s training pro- 
-ram* since the 1950s. when they had fixed-base instrument 
trainers in Chicago. His primary thrust since 1952 has been 
to advocate recurrent proficiency training. United started 
line-oriented flight training (LOFT) in 1976, and more 
recently has developed command, leadership, and resources 
(CLR) training. Dave has a group ot about 35 people includ- 
ing Flight Standards instructors and analysts to develop and 
execute training programs for all six aircraft that United 

operates. 


loft training 


Flight simulator currency training (recurrent proficiency 
checks — PCs) can be divided into “batting practice and 
LOFT. Batting practice represents the traditional approach - 
successive approaches, departures, failure modes and the like, 
not in a trip context. Dave emphasized that they do not slew 
0? restart the simulator at initial conditions (the beginning of 
an approach), but fly it around the pattern. Flying large jet 
aircraft requires staying ahead of the aircratt and anticipating 
future events. He felt that pilots lose the context and pacing 
of normal operations if they are slewed around to the start ot 
an approach. [Note this is a different position than was taken 
in the initial B-767 Computer-Based Training system by 
others at United, but the automated B-767 training program 
did not materialize . primarily because of a last-minute change 
from three-man to two-man operations and the impact on 
the simulator redesign, delivery schedule, and costs.] 

LOFT is well known as training in the context of a line 
trip from point A to point B. LOFT objectives are to provide 
training that combines the aircraft, the route, and crew inter- 
actions within cockpit and between cockpit and all external 
systems such as dispatch, ground crew, and ATC. LOFT pro- 
vides all normal trip activities, including trip paperwork, lor 
the whole crew. It permits crew interactions and exposure to 
past and recent line operational problems. Given the proper 
selection of valid United line trips, many of the problems 
which really occur can be built into the scenario. Dave claims 


4 Now retired from United Airlines. 


that if they are given more time, they could work all of the 
batting practice drills into a LOFT in certain areas, such as 
Southern California, Chicago-Cleveland-Pittsburgh, and 
Boston-New York-Washington. Current proficiency checks 
have two segments of LOFT in addition to the customary 
Batting Practice. 


CLR TRAINING 


There is heavy emphasis on CLR training because 80% of 
the accidents have nothing to do with the aircraft, but with 
“human factors,” which also used to be called “pilot error. 
Dave stated it was common knowledge that prior to CL , 
First Officers would become Captains on the basis ot time 
alone (seniority) and flying proficiency (being able to pass 
the checkride). They were thrust into command and 
decision-making jobs without the benefit of command and 
leadersliip training. He also observed that the need for such 
training was not limited to Captains, because the whole crew 
operates the aircraft, and must function efficiently and 

S3 f c ly 

CLR training is composed of academics (learning the 
theory from text and workbooks), seminar discussions, and 
specially designed and debriefed LOFT exercises. CLR 
emphasizes the concepts of a) Inquiry, b) Advocacy, c) Con- 
flict Resolution, d) Decision-Making, and e) Critique as e c- 
ments in the identification and resolution of operational 
problems in the cockpit. The approach uses the Blake and 
Mouton (1978) Grid to identify individual pilot styles. Blake 
and Mouton classify individual behavior in terms of either a 
basic task or people orientation. They quantify these two 
attributes in an x-y matrix scaled from 1 to 9 on each axis. 
Pilot styles can be identified and scaled along these dimen- 
sions. Dave commented that the military (or captain is 
king”) style docs not promote the best use of cockpit 
resources; actually. Dave was more emphatic when he said, 
‘it doesn’t work.” 

The key elements of CLR training are as follows: a) there 
is no single solution to the problems given, b) there is no 
interference by instructors, and c) there is no performance 
assessment by instructors. In a typical LOFT exercise for 
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CLR training, a major problem, which has a cascading effect 
will be introduced early in the flight; problems such as major 
hydraulic system failures are used. Minor “mosquito bite” 
type problems such as light bulb failures are avoided. 

There may be more than one problem, but they typically 
do not make the scenario too complicated; the normal con- 
straints of (light and the ways crews interact to solve prob- 
lems create a fertile environment for CLR training. The flight 
is permitted to develop naturally. No one interferes, stops 
the flight or tells the crews what to do. All scenarios are 
possible to execute safely, but if the crew makes too many 
mistakes, or a critical mistake at a wrong time, a crash could 

result. A crash is permitted, but Dave said they have never 
had one. 

United always provides at least two viable courses of 
action, so that the crew will be forced to make decisions, 
and will be able to use innovative problem solving techniques 
instead of being led down the path of least resistance by the 
constraints of the scenario. This echoes concerns that some 
scientists have had to permit truly emergent behavior to 
untold in team training situations (Crowe et al„ 1981). 

The crew is video taped, and all conversation and com- 
munications are recorded. At the conclusion of the flight, the 
video tape is taken to a closed room for crew review and 
debriefing. The instructor serves as a facilitator only; he does 
not evaluate or offer comments. He leads the discussion and 
focuses it to particular parts of the flight that the crew 
should review and critique for themselves. At the conclusion 
of the debriefing, the video tape is erased. 


COMMENTS 


IS 
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• Weather and effects on takeoff, enroute, fuel, and 
landing requirements. 

United derives data from accident reports, from irregular- 
ity reports within the company, and there is a “network” 
among the airlines and equipment manufacturers to share 
operating problems and solutions. United trains about 6,000 
pilots a year. Their instructors receive constant feedback on 
what happens on the line and, of course, they gain informa- 
tion on potential operating problems in the simulator train- 
ing sessions. In addition, there are line-check pilot reports 
and quarterly flight standardization meetings. In short, there 
is a constant flow of information on equipment, mainte- 
nance, ATC, airport, dispatch, route, and ciew difficulties. 
The severity and implications of problems (most of which 
actually have occurred in line operations) along with a j 
judgment of the ability of training to mitigate the problems, 
drives the selection of problems for LOFT or CLR training. 

Using these data, Dave Shroyer, who has more than 30 yr 
of experience with what does and docs not work in training, i 
collaborates with an aircraft fleet representative and assis- ; 
tant. United has six aircraft types. A fleet representative and 
assistant represent the technical expertise on each aircraft, j 
Dave frames the problem generally, the fleet representative 
and assistant write the training objectives, and Dave reviews 
their work as a quality control check. Thus, three people are 
directly involved in problem selection and scenario definition 
at the level of the training objectives. It must be remem- 
bered, however, that problem selection is based on a cafeteria 
of data and information which has been derived from all the 
sources which were described above. 


Problem Selection 

For LOFT and CLR, United looks for problems which are 
realistic, solvable, and have multiple implications for the 
remainder of the flight. As said before, they avoid “mosquito 
bite” problems, and look for those which will tax the team- 
work, system knowledge and decision making capability of 
the crew. They choose problems which will cause the crew 
to think ahead, perhaps to the approach and landing, and to 
plan what they would do if there are further problems, such 
as other failures, a change in weather, or a change in the 

landing runway. Problems can include but are not limited to 
the following types: 

• Electrical, hydraulic, and mechanical system failures. 

• Flight paperwork errors, dispatch procedures, weight 
and balance. 

• Crew or passenger problems (including bomb and 
hijack threats). 

• Problem ATC, noise abatement, or obstacle clearance 
procedures. 

• Problem airports, landing runways, and traffic delays. 


Scenario Construction 

About 4 wk is allocated to the construction of a new 
scenario and three or four people are involved. This time 
allocation assumes that the scenario analyst knows the air- 
craft and systems very well, the normal and emergency pro- 
cedures for the aircraft, the flight operations procedures used 
by United from dispatch to arrival at the gate, and the details 
of the specific departure airport, the route, and all possible 
terminal areas and airports. ! 

The estimated level of effort assumes that all the required 
data for frequencies, facility locations, terrain and airport 
models reside in the simulator data base. It also assumes a 
knowledge of candidate problems to give the crew and the 1 
training objectives that are addressed by those problems. f 
Dave is hesitant to guess what level of effort would be ! 
required to build a scenario from scratch without all this i 
institutional memory. He believes at least 6-8 person months ’ 
would be required, and possibly more, depending on the ; 
experience of the scenario development team. 

Once the departure airport, route, and nominal destina- [ 
tion are determined, the actual line route is observed by one ! 
of the scenario designers (in the jump seat), and all radio and ! 
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intercom communications are recorded from pre-push back 
to arrival at the gate. Air traffic control personnel at 
departure, enroute, and arrival locations are consulted. This 
is done to ensure that the actual language, procedures, and 
How of events of each route are as realistic as possible, and 
up to date. The scenario is then built and reviewed by Dave 
to ensure consistency, sufficient pacing of events, meeting of 
training objectives, and realism. If the training is to be certi- 
fied, the FAA has to approve it. 


Scenario Testing 

About a week is devoted to preliminary scenario testing 
to be sure that it works in the simulator. Flight Standards 
instructors at the training center review the scenario, botr 
from an execution viewpoint in the simulator and to test t re 
utility of the scenario for its training purpose. After prelimi- 
nary testing, the scenario is modified as necessary during 
initial instructor training. Scenario testing continues and it 
may be revised further after the first crews fly it. Flight 
crews often point out improvements in young scenarios. 
Scenarios are not considered to be debugged well or rela- 
tively stable until they have been used for about 6 mo. tven 
after that time, they have to be changed if there are any 
changes in the routes, frequencies, facilities, or procedures. 


Instructor Training 

Instructor training to administer a scenario requires about 
as much time as building the scenario initially. The nominal 
time is about 4 wk. United instructors maintain currency in 
all positions on the aircraft they are training, and maintain 
currency on more than one aircraft. They have a reservoir of 
institutional memory of prior scenarios and the general prin- 
ciples of LOFT and CLR training. Instructor training time 
could easily quadruple the current time allocated if this level 
of currency and institutional memory were not available. 


Cabin-Crew/Flight-Crew Interactions 

Communications with the cabin crew can be simulated 
easily, as can communications with the ground crew, 
necessary. United will simulate sending a crew member ‘ out 
of the cockpit” if the problem dictates. The rationale is that, 
for the training problem (and in reality), that person is 
simply not in the cockpit during that time. He or she will 

return and make a report. 

For the crew remaining in the cockpit, the illusion o 
Hight is preserved. For the person who “leaves” the cockpit, 
the illusion may be interrupted, but he or she is doing what 
they would normally do - leave the cockpit to do something 
and report back. The overt behavior is consistent with 


reality United doesn’t seem to be concerned with whether or 
not the illusion is maintained for the person who “leaves 
the cockpit. The problem itself is real, and the solution 
provides crews with the experience of dealing with it, which 
meets United’s training objectives. 


Stress and Peer Pressure 

As for stress created by peer pressure, it was observed that 
most professional pilots are quite sensitive about their perfor- 
mance. It was doubtful that being in a simulator with other 
crew members and an instructor would change this source ot 
stress from what normally exists in the real world. Since 
United is dealing with proficiency checks which have a direct 
bearing on continued employment, it is possible that there is 
more stress in their simulators than on a normal line flight. 


Organizational Pressure 

As one approaches LOFT-type scenarios and is investigat- 
ing the crew interactions and decision making that might 
occur in the real world, there are organizational pressures 
that might influence behavior in the real world, but may or 
may not influence behavior in the simulator. 

For example, if one declares an emergency, a report has to 
be written and the problem becomes known to many people 
in the company hierarchy. Some pilots may not want the 
hassle of the report, being “second guessed ’ by someone 
who was not there to see the whole situation untold, or 
having their name associated with an operational problem, 
however mundane. As another example, there arc difficulties 
for the company and the passengers if an aircraft does not 
land at the intended airport, or one which can handle the 
flight maintenance, or passenger requirements. 

Pilots are trained throughout their careers to maintain a 
reasonable margin of safety ; but it is seldom that an aircra t 
is dispatched in perfect working order, that all facilities along 
the route are operational, or that the weather is certain. 
Pilots have to make judgments. There is pressure to make 
each flight as economical as possible, and there arc relatively 
few absolute criteria. Pilot behaviors, and especially their 
decisions, will be conditioned by organizational pressure in 
the real world. The extent to which this behavior is exhibited 
in a simulator is unknown, but probably varies from pilot to 

pilot. , , , 

Dave Shroyer commented that United has not begun to 

address pilot judgment and decision making directly. Instead, 
they develop scenarios which will challenge the flight crew 
teamwork and decision process, let the flight unfold without 
interference, and guide the flight crew in their own critique. 
No judgments of the goodness or badness ot the performance 
arc made by the flight instructors during CLR training. 


OWQfaAL PAGE fS 
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Nonpilot Observers 

Wc asked Dave if a researcher could see what a skilled and 
experienced pilot would see while observing a simulator 
Hight. He commented that one designs a simulator night with 
some purpose in mind, and it doesn’t take a skilled and expe- 
rienced pilot to determine whether or not the desired behav- 
iors resulted. Observers have to be trained what to look for, 
and they may need training to increase their observational 
skill for a given environment and situation. Undoubtedly, 
night training and experience would help the observer under- 
stand the environment and what to look for, but, given some 
training, it was not essential. Dave commented that one of 
his scenario analysts is a psychology major, but he designs 
very good scenarios and understands the cockpit environ- 
ment very well. 


Fidelity 

Wc had to ask the obvious question: Can some or much of 
this training be done without expensive, high fidelity simu- 
lators with visual and motion systems? Dave thought that 
much training could be done with lower fidelity devices. He 
pointed to successes in training during the 1950s with much 
less simulator capability and fidelity, and to their whole 
training program, which includes all media. He commented, 
though, that the airlines are driven by requirements of the 
regulatory agencies, and by the legal implications of what 
they do. Having achieved zero-time (flight time) training and 
its cost benefits, it is unlikely that any airline would change 
anything that might jeopardize the benefits of the whole 
approach. This includes using the maximum state of the art 
in flight simulators. 


Validity of Behavior in Simulators 

Wc asked another point-blank question: Is the behavior 
you have observed of pilots in a simulator any different from 
what you would expect in the real world? Dave seemed sur- 
prised that anyone would ask this question. For him, there 
is no question that the behavior in their simulators is valid. 
He cited an example from the days before modern simula- 
tors, visual, and motion systems, where there was an inten- 
tional gear-up landing in Los Angeles caused by a system 
failure. The pilot did everything perfectly, and commented 
that it was “a piece of cake” because he had just practiced 
that problem in the simulator. 

This view probably is common in a commercial airline 
training environment, where there is high motivation to learn 
and maintain proficiency. We do not know if the same moti- 
vation would operate in a research simulator setting, but cer- 
tainly elements of professional pride in proficiency and 
previously mentioned aspects of peer pressures would be 
operating. Together they might create motivational levels 

which are equivalent to those found in commercial airline 
training. 


LOFT Guidelines 

We asked if there was anything in the LOFT guidelines 
report (Lauber and Foushee, 1981) that is no longer true, or 
is out of date. Dave said the principles are just as valid today 
as they were then. He knew of nothing that needed to be 
changed, and assured us that if there was anything controver- 
sial in that report, he would certainly be aware of it He was 
most pleased that the Air Force Military Airlift Command 
has pursued the development of LOFT exercises under the 
rubric of MOS training (MOST) based on the LOFT guide- 
lines report. 
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APPENDIX I 


FIELD INTERVIEW 

McDonnell douglas corporation military mission-oriented 

SIMULATION RESEARCH 


Dr. William J. Cody 
Lead Engineer, Life Sciences Division 
McDonnell Douglas Corp., St. Louis, Missouri 


Dr. Cody directed three major studies for the Air Force 
involving use of the McDonnell Douglas flight simulator 
facility. Two studies had the purpose of quantifying the 
effects of chemical defense (CD) stressors on pilot perfor- 
mance. The third was an evaluation of the mission effective- 
ness of an F-15 Dual Role Fighter (DRF) crew system con- 
cept for a pilot and weapons systems officer. All three 
studies involved air-to-ground attack scenarios. 


CUSTOMER RELATIONS 


The customer imposed difficult constraints on the CD 
studies. Only six pilots were to be tested because of cost, and 
the collection of baseline, i.e., normal operation, data was 
not included because this was in effect a test of the pilots 
competence. 

Discussions with the customer to translate the customer s 
requirements to testable propositions is an important part of 
the initial work. The customer almost always formulates the 
study question in a general or practical way. It is not always 
clear what the research questions should be to answer the 
practical question. The researcher should be careful not to 
undertake the project without a clear understanding with the 
customer of what will be done in specific terms. 


SCENARIO DEVELOPMENT 


The main issue in the chemical defense study was how- 
body heating equivalent to wearing individual protective 
equipment (IPE) would affect attack-mission performance. 
The heat load was imposed by an undergarment with tubing 
woven in for circulation of hot water. It was difficult to 
know how long the segments should be. The customer pro- 
vided data on what the expected body heat change should 
be with time wearing the IPE, but it was not possible to 
determine experimentally ahead of time what temperature 
profile for heating the suit would produce the desired body 
temperature, or how long the mission segments should be. 


In the first CD study the mission profiles originally con- 
ceived turned out in preliminary testing to be far too diffi- 
cult. They had to be made less demanding to be practical. 
(Notice that this problem is common to the predisposition 
to overcomplicate LOFT or LOS scenarios.) Six different 
profiles were developed for the first CD study. These profiles 
were considered initially to be equally difficult based on the 
total distance flown, the number of waypoints and the num- 
ber of heading changes. It turned out, however, that pilots 
found them to be very different in difficulty because of dif- 
fering demands associated with angles to the target, distance 
and the weapon used. Cody said that his lack of familiarity 
with the details of the attack mission led him to initially 
oversimplify the equal-difficulty problem. 

In the second CD study a different approach was taken to 
develop equivalent profiles. A single mission that had several 
legs, waypoints and two target locations was designed. A 
template of this profile was drawn and other profiles wore 
generaied by rotating the template with respect to the simu- 
lated terrain. Thus the specific headings, terrain path and tar- 
get locations all changed, but the fundamental profile did 
not. Since the pilots think and perform in terms of the spe- 
cific headings and ground references, they did not realize 
that the different missions were fundamentally the same. 
This was a successful means for creating differences in the 
appearances of the scenario, while maintaining the similari- 
ties necessary for data analysis. A similar technique was used 
in the DRF study. 

In the DRF study, relatively short mission segments were 
used. The scenario began approaching the forward edge of 
the battle area (FEBA) and ended on the return crossing. 
Cody found that his experiences performing the CD studies 
were a great help in performing the DRF study. Length and 
character of the scenario were dictated by a consideration of 
the number of observations, i.e., data collection segments 
necessary and the crew functions to be performed. Tasks 
were segmented and approximate performance times were 
associated with each task segment. Laying out the segments 
on a time line and taking data collection needs into account 
produced an estimate of the length and composition of the 
scenario. 

The simulation for the CD and DRF study were not full 
mission in the sense of including every aspect of the mission. 
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Compromises were made in the interest of economy of effort 
and experimental control. For instance, the mission planning 
phase, which can require 2-3 hr for a 1 -hr mission, was not 
included. Also, communications were restricted because it 
was not clear how they could be controlled and manipulated 
in a way advantageous to the study. They would be a con- 
founding factor only, and not essential to the flying of the 
mission. Only the systems essential to the mission were oper- 
ational. This saved training time and removed additional 
sources of variability. Also, cockpit checklists were abbre- 
viated to include only those systems that were essential to 
performing the mission. These were all acceptable reductions 
in mission fidelity, because the purpose of the studies was to 
determine how well the pilots could perform specific tasks 
with particular equipment and procedures. 

For the CD studies, four different models were used to 
predict what behavioral functions related to mission perfor- 
mance would likely be sensitive to heat stress. The predic- 
tions from these models were used as a basis for the scenario 
design, i.e., to emphasize aspects of the mission that are most 
likely to show effects. For example, tracking performance 
was one of the functions predicted to be sensitive to thermal 
stress. Consequently, the scenario was designed to include 
tracking segments with wind-gust disturbances. Pilots 
thought these relatively long straight segments were unrealis- 
tic, but realism was compromised for the sake of obtaining 
useful data. 

F-15 pilots were used in the preliminary testing of the 
DRF missions. Several changes to the scenario were made 
based on their comments. This reinforces the point that pre- 
liminary testing is a critical part of any simulation-based 
research. 

To keep the scenarios simple, Cody had originally planned 
to use a single weapon type. The pilots objected because they 
said that would never be done. So to satisfy the pilots in the 
DRF study, appropriate weapons were paired with different 
targets although it was not a factor of interest in the 
research. 


SIMULATOR ISSUES 


The simulator facility is essentially modular in hardware 
and software. Developing the simulation thus involved link- 
ing the hardware units necessary, e.g., the cockpit and terrain 
board, and then assembling the proper software elements. 
Problems in the development of the simulation configuration 
included taking real-time computational demands into 
account. The number and type of events, as well as the data 
logging needs, had to be assessed to ensure the computer 
would not be overloaded. The parameter definitions and for- 
matting also had to be done with care. Great detail of specifi- 
cation was necessary to communicate to the programmers 
exactly what the investigator’s intentions were. 


In both studies a constraint was imposed by the simulator 
visual system 60 field of view and the amount of terrain 
available on the terrain board. It took quite a bit of effort 
to develop a profile that would keep the target in the field of 
view long enought to perform the attack run, and still keep 
the visually guided flight portions of the mission within the 
limits of the terrain board. 

Shortcomings of the simulation for the purpose of the 
studies included the inability to program event-versus-time- 
contingent occurrences, the relative inflexibility of the code 
(it takes a great deal of programming effort to change a pro- 
file), and the lack of real-time data reduction to provide sum- 
maries of a run shortly alter they occurred. It was vital to 
keep a test director’s log to record the time of start and end- 
ing of segments of interest for data analysis, and to note 
when some problem or other produced bad or contaminated 
data. 


SUBJECT ISSUES 


Selection and training 

Three types of aircrew members were used in the DRF 
study. Three crews had experience in the two-seat F-15B2 
aircraft. The other three crews were composed of three F-15 
(single seat) pilots and three F-l 1 1 weapons systems officers. 
The experience of three different types of aircrew personnel 
affected the comprehensiveness of the simulation and the 
choice of tasks. A practical by-product of using the F-15B2 
pilots was their exposure to the system configuration devel- 
oped by the company. To the degree that their experiences 
in the simulator gave them an appreciation of the merit of 
this configuration, they would become positive advocates for 
it in the operational community, it, of course, made sense to 
use these pilots for the study because it minimized the train- 
ing problem. 

Issues about the interaction of subject characteristics, 
mission task requirements, and simulator fidelity became 
apparent in the CD studies. Two types of pilots were used, 
F-15 pilots and Air Guard A-7 pilots. An F-15 simulator was 
used, and the task was air-to-ground weapon delivery. F-15 
pilots are familiar with the F-15 cockpit, but because this 
aircraft has an air-combat mission, the pilots were not profi- 
cient in ground-attack maneuvering. The A-7 pilots were 
unfamiliar with the F-15 cockpit, but knew ground-attack 
procedures very well. The trade-offs were to teach F-15 
pilots to do ground attack, and familiarize the A-7 pilots 
with the F-15 cockpit since both types of pilots were used, 
the F-15 pilots in the initial study and A-7 pilots in the sec- 
ond. When the results of the two studies were compared, it 
turned out that the A-7 pilots were about twice as good in 
their bombing scores as the F-15 pilots. 
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Normally, subjects are chosen because of their familiarity 
with both the specific aircraft and the mission. In this case 
one group was familiar with the aircraft and the other with 
the mission. The experience here was that, in terms of mis- 
sion performance, it was easier to teach the A-7 pilots to fly 
the F-15 than to teach the F-15 pilots to perform the mis- 
sion. However, had the mission depended heavily on use of 
all the aircraft systems, it is likely that the F-15 pilots would 
have done better. The lesson is that choosing the subject 
population is not always a simple decision. Specific experi- 
ences must be weighed against mission requirements as well 
as the configuration of the simulation equipment. 

In training the pilots for the attack mission, a profile that 
was one of the variants of the general mission profile was 
used. The criterion for completion of training was three suc- 
cessful weapons delivery runs. 


Physical and Psychological Well-Being 

When research involves physical stressing of the subject, 
the performing organization should not accept responsibility 
for the well-being of the subjects. In this case, the Air Force 
provided a medical doctor to monitor the tests and accept 
responsibility for the Air Force pilots. Understandably, in 
studies where behavioral changes or physiological effects are 
expected to occur, it is difficult to get informed consent 
from pilots. 

Surface-to-air missile (SAM) threats were included in the 
mission profile. An issue was whether to allow the aircraft 
to be hit by a SAM and, if so, would the mission stop. It was 
essentially a realism vs. research practicality issue. If the 
aircraft was never hit, the pilots would soon learn that the 
missiles were really not a significant threat. Since these are 
operational pilots, there was a danger that altering their 
expectations of the real threat could have lethal implications. 
On the other hand, it would disrupt the research if the simu- 
lation terminated when the aircraft was struck by a missile. 

The compromise was to allow’ the aircraft to be hit if the 
pilot did not counter the threat by ECM or maneuvering, 
but only cue the pilot that he was hit and not stop the mis- 
sion. Unlike civilian air operations, military pilots are used 
to disasters, i.e., dying in mock combat. They do not like it, 
but do see it as a valuable training experience. Because this 
is routine in military training, it does not have the ethical 
implication of possible psychological harm to the pilot, 
which would be the case in simulations of civiTair operations. 


Subject Attitudes 

F-15 pilots were reluctant to participate in the CD study 
because they did not want to give up aircraft flight time. The 
Air Guard pilots were more willing to participate because of 


the opportunity to get some F-15 experience, if only in a 
simulator. 

Txperienced pilots were used in the CD and DRF studies, 
and are used in most of the simulation work McDonnell 
Donglas does. The cockpits are comprehensive and realistic 
representations of the actual aircraft, and t he aircraft dynam- 
ics arc accurate. (The F-15 and F-18 design was basically 
derived from simulation developments rather than the 
reverse.) Physical fidelity is not an issue in the simulation 
studies. Pilots do express some concerns about the lack of 
cockpit motion because the main simulators are all fixed 
bast . The pilots had no concerns about the low luminance of 
the visual simulation, but were unhappy with the limited 
resolution and field of view. Early target detection and side 
viewing are important to air-to-ground attack missions and 
are, therefore, the likely basis of the concerns about resolu- 
tion and field of view. 

The main concern of the subjects is mission or scenario 
fidedty. Pilots do not readily accept deviations from opera- 
tional practice unless the study is clearly to try new equip- 
ment and procedures. Part of the concern about scenario 
fidelity stems from the fact that a simulation that is essen- 
tially a duplication of an actual mission is a test of the pilots' 
capabilities. If the pilots are to submit to such testing, they 
want everything right to maximize their opportunity to prop- 
erly perform. They do not want a shortcoming in the simula- 
tion to be interpreted as a lack of their ability. Conversely, 
the pilots are much more favorably disposed to trying new 
equipment and procedures because they cannot be held 
aect untable for the outcome. 

Cody commented that the pilots are very skeptical that a 
non pilot psychologist can measure expert pilot behavior. It 
is important to include a pilot on the research team to work 
witl the researcher and interact together with the subject 
pilots. The subjects then have some assurance that somebody 
who knows the operational world is involved, and can discuss 
the reasons for the characteristics of the simulation study in 
credible terms. 


PERFORMANCE MEASUREMENT 


Multiple means of performance measurement were used 
by Cody on the CD and DRF studies. Expert opinion of 
experienced pilots was used heavily during the scenario devel- 
opment and checkout phases. Informal dialogue between the 
^experts and the investigator were the primary means used in 
these stages. The Subjective Workload Assessment Technique 
(SWAT) was administered routinely to the subjects since 
workload was a major consideration in both studies. 

The subjects also were given questionnaires before and 
aftci the test sessions. The preliminary questionnaires collect 
biographical and experience information. The post-test ques- 
tionnaires were another source of expert-opinion information 
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about the simulated system and procedures, as well as the 
simulation and scenario. The pilots also were asked to rate 
their own performance. 

Automatic recording of system-state data was used exten- 
sively. It was the primary source of objective data, and was 
relied on for final analyses as much as possible. The general 
philosophy was to collect as much of this type of data as 
possible; it may turn out to be useful later. Video and audio 
tapes were valuable sources of information. Crew actions, 
communications, cockpit displays and mission track were all 
recorded. These were later edited together in a time-linked, 
split-screen format to simplify interpretation. These tapes 
were used for link-analysis, classification of crew communica- 
tions and observer scoring of performance. 


SCHEDULING 


Cody noted that scheduling of simulator time was a prob- 
lem and they usually worked off-hours. An informal priority 
of simulator use is followed. First priority is marketing, 
second is engineering development of specific aircraft sys- 
tems, third is training company pilots, and fourth is con- 
tracted research. 

The initial schedule for data collection was too tight. He ; 
tried running three pilots or crews per week, but simulator 
failures and subjects not showing up on time created severe, j 
sometimes impossible, problems for completing the planned { 
runs. Cody said that during the period of actual data collec- } 
tion, the investigator should plan on having about 50% of \ 
the scheduled simulator use time being used for the runs. \ 
That is, if the simulator is scheduled for 8 hr/day, no more 
than 4 hr/day of actual running time should be planned. ( 
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